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FOREWORD 


This  guide  is  intended  to  serve  as  an  aid  for  training  developers,  training  reviewers, 
training  implementers,  and  training  evaluators.  It  was  designed  to  address  a  broad  range  of 
training  development  projects  that  all  fall  under  the  general  heading  of  “structured  training.” 

All  of  the  work  that  led  to  the  codification  of  this  methodology  was  performed  under  the 
sponsorship  of  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI), 
Armored  Forces  Research  Unit  (AFRU),  Fort  Knox.  Three  separate  but  related  training 
programs  and  the  associated  training  support  packages  (TSPs)  have  been  or  are  being  developed 
using  the  methodology; 

•  Combined  Arms  Operations  at  Brigade  Level,  Realistically  Achieved  Through  Simulation 
(COBRAS)  —  brigade  staff  exercises  and  brigade  staff  vignettes, 

•  the  Virtual  Training  Program  (l^TP)  —  exercises  for  platoons,  companies,  battalions, 
battalion  staffs,  and  brigades,  and 

•  Structured  Training  for  the  Close  Combat  Tactical  Trainer  (CCTT)  —  exercises  for 
platoons,  companies,  and  battalions  using  CCTT. 


Users  are  encouraged  to  “color  outside  the  lines.”  If  the  methodology  does  not  address  a 
particular  need,  then  the  creative  developer  will  find  a  solution.  But  it  is  important  to  know  what 
some  of  the  lines  are.  This  guide,  in  prescribing  various  development  considerations  and 
procedures,  defines  many  of  the  important  issues  and  design  challenges  that  developers  must 
confront  and  resolve. 
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GUIDE  TO  DEVELOPMENT 
OF  STRUCTURED  SIMULATION-BASED  TRAINING 


This  figure  presents  an  overview  of  the  methodology  that  will  be  described  in  this  guide. 

Key  features  include  the  four  phases  of  development  and  the  continuous  nature  of  the 

formative  evaluation. 

The  figure  is  explained  on  the  next  page. 


PHASES  OF  THE  DEVELOPMENT  METHODOLOGY 


FORMATIVE 

EVALUATION 


Phase  1: 
Initial  Decisions 


Phase  3: 

Design  Scenario  and 
Exercise  Outline 


Decisions  on: 

•  Target  audience 

•  Training  context 

•  Simulation  technology 
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Figure  1 .  Overview  of  the  development  methodology  for  structured  simulation-based  training. 
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GUIDE  TO  DEVELOPMENT  OF 
STRUCTURED  SIMULATION-BASED  TRAINING 


INTRODUCTION  TO  THE  GUIDE 
The  Four  Phases  in  Development 

There  are  four  phases  to  this  development  methodology,  as  shown  in  Figure  1  on  the 
previous  page: 

•  Phase  1:  Initial  Decisions — Determine  the  training  requirement  (e.g.,  mission  and  enemy 
type,  terrain,  time  constraints,  number  of  exercise  start  points,  difficulty  level),  training 
audience  (e.g.,  unit  type  or  echelon,  personnel  within  unit),  and  appropriate  training 
environment  (i.e.,  specific  simulator/simulation). 

•  Phase  2:  Select  Training  Objectives — Focus  the  training  on  critical  tasks  and 
performance  standards  in  support  of  the  training  requirements,  and  ensure  that  those  tasks 
can  be  performed  in  the  selected  simulator/simulation  environment. 

•  Phase  3:  Design  Scenario  and  Exercise  Structure — Determine  the  limits  of  each  exercise 
with  reference  to  METT-T;  generate  the  tactical  framework  for  the  exercises;  specify  the 
events  within  each  exercise;  define  and  put  substance  to  the  roles  to  be  played  by  the 
simulation  and  by  various  types  of  trainers;  and  crosswalk  the  training  objectives  to 
events  within  the  exercises. 

•  Phase  4:  Develop  Training  Support  Package  (TSP) — Construct  and  try  out  all  of  the 
written  and  simulator/simulation-based  components  of  the  training  program,  including 
materials  for  the  trainers  and  for  the  participating  unit. 

These  four  phases  need  not  be  done  in  a  strictly  linear  fashion.  The  continuous  attention 
to  formative  evaluation  during  an  activity  will  often  cause  you  to  go  back  and  revise  the  products 
of  an  earlier  phase.  In  fact,  you  should  regard  the  entire  process  as  flexible,  and  be  prepared  to 
move  back  and  forth  between  phases  during  the  development,  revising  decisions  and  products  as 
necessary.  For  example,  understanding  of  the  initial  decisions  will  change  as  more  is  learned 
about  the  simulation  and  the  tasks;  information  that  feeds  into  development  of  the  TSP  will  be 
captured  during  documentation  of  initial  decisions;  and  so  on. 


Whenever  decisions  or  products  are  revised,  you  must  trace  back  through  earlier 
development  and  correct  all  related  products,  whether  they  are  interim  products  or  parts  of  the 
final  TSP.  We  cannot  emphasize  too  strongly  the  importance  of  keeping  all  products  current  and 
in  agreement  with  each  other.  We  should  also  point  out  that  this  is  one  of  the  most  difficult 
challenges  in  development. 
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The  Nine  Activities  in  the  Four  Phases 


There  are  nine  Activities  within  the  four  phases.  In  the  guide,  each  phase  of  the 
methodology  is  introduced  by  a  discussion  of  the  purpose  of  that  phase,  along  with  a  list  of  the 
activities  in  the  phase.  Each  activity  is  then  described  in  detail.  Where  it  seems  helpful, 
examples  from  recent  development  efforts  have  been  included. 


Figure  2  shows  the  activities  in  each  of  the  four  phases. 


ACTIVITIES  IN  THE  PHASES  OF  DEVELOPMENT 


Formative 

Evaluation 


Phase  1: 
Initial  Decisions 


Phase  2: 

Designate  Training 
Objectives 


Phase  3: 

Design  Scenario  and 
Exercise  Outline 


Phase  4: 
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Package  (TSP) 
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Activity  2.1:  Identify  Task  Sources,  Tasks,  and 
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Activity  2.2:  Refine  Task  List  for  Simulation  Support 
Activity  2.3:  Select  Tasks  That  Support  Mission 
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Activity  3.3:  Outline  Events  and  Build  Exercise 
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Structure 

Activity  4.2:  Prepare  Training  Support  Package  (TSP) 
Materials 


Figure  2.  The  nine  activities  in  the  four  phases  of  the  development  methodology. 


Although  the  guide  presents  a  succinct  procedure  for  developing  structured  simulation- 
based  training,  the  guide  alone  will  not  be  sufficient  for  development.  Subject  matter  expertise 
for  the  selected  technology,  for  associated  military  aspects  (e.g.,  the  mission  type,  xmit  type, 
operations,  and  enemy  tactics),  and  for  training  and  instructional  design  will  be  required 
throughout  the  development  and  review  processes. 


The  Formative  Evaluation 


The  methodology  incorporates  a  series  of  required  and  critical  formative  evaluation  steps 
throughout  the  development  process.  Some  of  these  are  formal  (such  as  map  exercises  conducted 
by  the  design  team,  tryouts  with  representative  or  surrogate  units  and  individuals,  technology- 
driven  tryouts,  or  content  reviews  with  experts  and  stakeholders  or  proponents),  but  formative 
evaluation  also  includes  a  continuous  attention  to  the  need  for  revisions  and  improvements. 

Their  purpose  is  to  insure  the  quality  of  the  product  by  attending  to  that  quality  throughout 
development;  they  are  intended  to  direct  attention  to  training  utility  as  well  as  technical  content. 

Figure  3  presents  an  overview  of  the  formative  evaluation  scheme. 


FORMATIVE  EVALUATION  IN  DEVELOPMENT 
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Figure  3.  Formative  evaluation  activities  in  the  development  of  structured  simulation-based 
training. 
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In  the  guide,  appropriate  formative  evaluation  activities  are  described  for  each  of  the 
developmental  activities.  Each  formative  evaluation  activity  has  a  specific  focus,  appropriate  to 
the  development  activity,  and  the  information  gathered  is  used  to  refine  products  to  ensure  their 
accuracy  and  usability  of  the  products.  The  formative  evaluation  activities  for  each  phase  of  the 
methodology  include  the  following: 

•  Phase  1:  Initial  Decisions — ^The  formative  evaluation  task  will  be  to  make  sure  that  all  of 
the  parameters  (the  “givens”)  of  the  proposed  structured  training  have  been  examined  by 
those  who  have  commissioned  the  training.  All  of  the  directives  that  have  been  given  by 
the  training  program  proponent  (the  office  or  individual  who  is  sponsoring  or  has  directed 
that  the  development  be  done)  should  be  documented  by  the  developers  and  approved  by 
that  proponent.  This  will  ensure  that  the  development  is  on  track  with  the  initial  intent. 

It  is  crucial  that  the  proponent  review  and  approve  of  any  changes  in  the  design  and 
development  work. 

•  Phase  2:  Designate  Training  Objectives — Developers  will  be  identifying  the  specific 
training  objectives  for  the  program.  These  are  the  tasks  that  the  program  will  address  and 
train,  and  the  standards  for  task  achievement.  The  formative  evaluation  in  Phase  2  should 
take  the  form  of  an  expert  review  of  the  selected  tasks.  This  review  should  involve  any 
offices  or  agencies  who  are  responsible  for  doctrine  and  documentation  for  the  subject 
matter  (e.g.,  doctrine  or  training  or  tactics  directorates).  These  agencies  have  a  vested 
interest  in  the  product  and  a  responsibility  for  its  accuracy;  they  are  also  knowledgeable 
and  can  provide  valuable  information  and  support. 

•  Phase  3:  Design  Scenario  and  Exercise  Structure — The  formative  evaluation  will 
become  more  active,  involving  map  exercises  and  simulation-controlled  exercises,  and 
finally  exercises  with  representative  (or  surrogate)  participants  (usually  referred  to  as 
“pilots”).  These  exercises  are  intended  to  evaluate  the  technical  content  of  the 
development,  with  regard  to  both  doctrine  and  simulation.  There  is  still  a  requirement  for 
proponent  and  expert  reviews  of  the  scenario,  to  ensure  that  the  evolving  design  is  still 
consistent  with  the  intent. 

•  Phase  4:  Develop  Training  Support  Package — The  formative  evaluation  will  involve 
representative  individuals  and  units  in  trying  out  the  program's  TSP  (referred  to  as 
“trials”).  By  this  time,  developers  should  have  performed  reviews  to  ensure  that  the 
exercises  are  doctrinally  correct  and  that  they  are  matched  to  simulation  capabilities,  and 
are  ready  to  check  on  how  usable  they  are  and  whether  training  occurs.  These  trials  are 
extensive  and  require  careful  planning,  rigorous  and  intensive  information-gathering,  and 
documentation  of  resulting  actions.  They  are  the  final  point  of  revision  and  require  close 
attention  to  detail  in  order  to  ensure  that  all  materials  are  complete  and  correct. 
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The  Methodology  Outline  (Appendix) 


The  outline  in  the  appendix  presents  the  four  phases  in  the  methodology,  and,  for  each 
phase,  the  activities  that  are  to  be  performed  in  the  required  analysis,  design,  and  development 
phases.  As  appropriate,  some  of  the  most  important  design  and  development  considerations  are 
also  listed.  For  each  of  the  activities,  the  outline  indicates  the  activity’s  product  or  outcome.  In 
many  cases,  the  activity  “product”  is  an  item  that  is  used  in  the  continuing  development  process, 
rather  than  a  piece  of  the  eventual  TSP.  The  outline  also  suggests  the  type  of  tryouts  and/or 
expert  reviews  to  conduct  as  part  of  the  continued  formative  evaluation. 

The  appendix  serves  as  a  job  aid  or  checklist  for  the  development  process  and  includes 
little  detail  or  explanation.  It  is  designed  to  help  maintain  high  quality  in  the  TSP  materials.  It 
lists  the  characteristics  of  correct  and  complete  TSP  materials.  Thus,  it  should  be  used  to  remind 
experienced  developers  and  reviewers  of  the  critical  elements  of  the  development  process  and  the 
TSPs. 


The  procedural  guide  to  the  methodology  for  developing  structured 
simulation-based  training  begins  on  the  next  page. 
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PHASE  1.  INITIAL  DECISIONS 


Activity  1.1  Document  initial  decisions. 


In  the  first  phase  of  the  methodology,  you  will  make  or  identify  certain  decisions  about 
the  training  that  is  to  be  developed,  and  then  document  those  decisions.  Whether  you  are 
performing  the  initial  needs  analysis  or  simply  recording  the  results  of  an  already-completed 
analysis,  it  is  crucial  that  those  initial  decisions  be  preserved.  Phase  1  has  only  one  activity — ^to 
document  the  initial  decisions. 


Activity  1.1  Document  initial  decisions 

Product/Outcome;  Documentation  of  decisions. 

Formative  Evaluation.  Proponent  review. 


The  decisions  that  you  make,  examine,  and  record  will  serve  as  the  basis  for  all  of  the 
other  activities.  These  decisions  should  arise  out  of  a  needs  assessment,  performed  by  you  or  the 
proponent  before  the  remaining  design  work  begins. 

There  are  at  least  four  areas  of  decisions: 

1 .  Training  Audience;  The  “training  audience”  is  the  personnel  for  whom  the  training 
is  to  be  designed.  In  some  cases,  this  will  be  the  entire  unit  at  the  selected  level  (e.g., 
armor  platoon);  in  most  cases,  the  training  is  intended  for  a  specific  subset  of  the  unit 
(e.g.,  battalion  staff).  This  may  seem  too  obvious  to  be  noted,  but  it  should  be 
documented,  to  ensure  that  all  parties  understand  the  training  focus.  In  general,  you 
will  be  developing  training  objectives  and  observation  protocols  only  for  members  of 
the  training  audience. 


EXAMPLE:  Designating  the  training  audience 

GIVEN:  A  requirement  to  design  staff  synchronization  and  integration  training  for  selected 
members  of  a  brigade  staff,  including  the  primary  staff  and  fird  support,  dir  defense  artillery, 
engineers,  and  combat  service  support. 

Developers  determined  that  the  pr/mary  training  audience  woulB  include  the  primary  staff  and 
those  individuals  who  rqpresentrthe  ofher  functions  to  the  brigade  commander.  Thus  for 
example,  the  Brigade  Engineer  vi/as  to  be  a  member  of  the  primary  training  audiende. 

The  next-removed  level  of  interaction,  those  whb  work  within  the  other  functions  and  the  staff 
..gggti9ng;..y!^4!!j..l?g..rgPrgggd^gd  by  thd  actuai  role  ihcumbentsrbut  they  would  hot  receive  the 


same  level  of  training  focus  (i.e.,  no  formal  observation  and  feedback  protocol).  These 
individuals  were  referred  to  as  ro/ep/ayers. 

Finally,  the  next  level  of  activity  would  be  represented  by  the  simulation  itself,  controlled  by 
trained  interactors.  Interactors  would  work  under  direction  of  roleplayers,  who  would  respond  to 
the  primary  training  audience  orders  and  provide  input  to  the  training  audience  as  requested. 


2.  Training  Context:  The  training  context  is  the  situation  or  scenario  that  will  provide 
the  “storyline”  in  which  the  training  is  set.  The  training  context  for  military  training 
programs  in  tactical  settings  will  likely  include  the  mission  type,  enemy  type,  terrain, 
and  unit  type. 

•  Mission  Type.  The  mission  provides  the  context  for  the  training.  The  mission 
type  will  affect  much  of  what  follows,  and  should  be  specified  early  in  the 
process. 

•  Enemy  Type.  Whether  this  is  an  actual  enemy  or  a  fictitious  enemy  force,  the 
enemy  needs  to  fight  according  to  defined  rules  and  tactics.  You  need  to  specify 
what  will  define  the  tactics,  doctrine,  organization,  and  equipment  for  the 
opposing  forces  (OPFOR).  If  the  OPFOR  is  to  represent  a  real  enemy,  then  the 
source  of  that  enemy’s  capabilities  must  be  identified.  If  the  OPFOR  will  fight 
according  to  an  approved  Training  and  Doctrine  Command  (TRADOC)  model, 
then  that  model  should  be  identified. 

•  Terrain.  The  terrain  setting  that  the  exercise  uses  should  be  indicated  in  terms  of 
the  general  area,  such  as  the  National  Training  Center  (NTC),  or  Fort  Knox,  or 
Korea.  Later  you  can  get  more  specific  about  the  precise  locations  where  events 
will  occur.  Insure  that  it  is  available  in  the  simulation. 

•  Unit  Type.  It  has  already  been  stated  that  the  training  audience  should  be 
specified.  Make  sure  that  you  consider  any  additional  defining  factors.  For 
example,  if  the  training  is  designed  for  brigade  staffs,  you  will  want  to  know  the 
type  of  brigade  (e.g.,  conventional  or  digitized,  armor  or  mech  or  light,  3-battalion 
or  4-battalion).  One  way  to  specify  the  unit  type  is  by  means  of  the  appropriate 
Tables  of  Organization  and  Equipment  (TOE)  designation. 

wr  NOTE:  Some  training  exercises  may  not  require  that  an  enemy,  terrain,  or  unit 
type  be  specified.  Battle  oriented-exercises  (i.e.,  those  that  require  the  unit  to 
fight  against  an  enemy  during  the  exercise)  will  require  a  well-defined  enemy, 
terrain,  and  unit  type.  But  exercises  that  work  on  staff  interactions,  non- 
battlefield  activities  (such  as  those  that  occur  in  assembly  or  rear  areas),  or  certain 
aspects  of  planning  processes,  may  not  require  such  a  fully  developed  mission, 
enemy,  troops,  terrain  and  time  (METT-T).  Examine  the  training  environment 
carefully  and  continue  to  examine  it  as  the  exercise  matures,  in  order  to  be  certain 
that  all  necessary  specifications  have  been  addressed,  decided,  approved,  and 
documented. 
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3.  Simulation  Technology:  If  there  are  different  versions  of  the  simulation  software, 
document  which  version  is  required  for  the  exercise.  Although  you  will  do  a  detailed 
analysis  of  the  technology  capabilities  in  Activity  2.2,  you  should  at  this  point  verily 
that  the  selected  technology  is  suitable  for  the  type  of  training  that  you  are  designing. 
For  example,  you  would  not  use  SIMNET  or  Janus  for  gunnery  training;  you  would 
not  use  mobile  SIMNET  for  brigade-level  training.  Additionally,  you  should  find  out 
as  soon  as  possible  what  version  of  the  simulation  technology  is  going  to  be  available 
for  the  intended  users  of  the  training. 

4.  Other  Considerations:  Depending  on  the  particular  development  project,  there  may 
be  other  considerations.  These  may  include  exercise  time,  number  of  scenario  entry 
points,  nature  of  entry  points,  linkages  to  other  programs,  particular  training 
emphases,  availability  of  trainer  resources,  and  so  on. 

•  Exercise  time.  Every  training  implementation  will  have  to  be  conducted  within 
certain  constraints;  time  constraints  are  the  most  immediate  and  apparent.  You 
need  to  identify  a  realistic  target  for  exercise  duration,  so  that  the  unit  can  use  the 
program.  This  estimate  of  available  training  time  allows  you  to  determine  the 
tasks  that  can  be  addressed  in  the  training. 

•  Number  of  entry  points.  Exercises  (and  TSP  materials)  that  are  structured  with 
multiple  entry  points  give  units  some  options  concerning  where  they  can  begin 
and  end  their  training.  However,  it  will  not  always  be  feasible  to  structure 
exercises  this  way.  If,  for  example,  you  are  designing  exercises  for  a  staff  or  staff 
section,  you  may  find  that  the  flow  of  events  is  so  continuous  that  it  is  extremely 
difficult  to  generate  all  of  the  relevant  cues  and  conditions  for  the  entry  point  and 
design  a  method  for  getting  training  participants  and  controllers  read  in  on  those 
conditions.  Thus,  starting  an  exercise  at  different  points  may  be  very  difficult  for 
both  the  training  audience  and  the  exercise  controller,  and  could  cause  an 
unacceptable  decrement  in  training  value. 

•  Nature  of  entry  points.  If  the  mission  is  to  have  multiple  entry  points,  it  should 
be  decided  whether  the  entry  points  will  be  based  on  the  unit's  expertise 
(proficiency-based)  or  on  training  emphasis  (needs-based).  Proficiency-based 
entry  points  require  that  associated  pieces  of  the  exercise  be  at  varying  levels  of 
difficulty.  Needs-based  entry  points  assume  that  the  exercise  segments  will  focus 
on  different  skills  or  activities.  The  determination  will  shape  certain  pieces  of  the 
TSP,  particularly  those  that  are  used  by  the  unit  to  select  its  training  options. 

•  Linkages.  There  may  be  a  requirement  to  link  several  training  programs  by 
putting  them  on  common  terrain,  making  them  part  of  a  higher  level  mission  or 
campaign,  or  making  them  represent  different  aspects  of  the  same  mission  (e.g., 
fire  support,  logistics).  Once  any  such  linkages  are  identified,  the  development 
efforts  should  be  planned  and  constructed  in  such  a  way  as  to  ensure  that  all  of  the 
related  efforts  can  satisfy  their  training  goals  while  satisfying  the  requirement  of 
commonality. 
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EXAMPLE:  Working  with  required  linkages^^^  ^  ^  ^ 

GIVEN:  A  requirement  to  design  training  for  battalions  that  used  the  same  scenario  (METT-T) 
as  previously  developed  platoon  and  company  exercises. 

In  order  to  establish  the  links,  developers  adopted  the  battalion-level  OPORD  that  had  been 
developed  as  the  overarching  context  for  the  company  and  platoon  exercises.  This  ensured 
that  the  mission,  enemy  intelligence,  terrain,  and  task  organization  would  be  congruent. 

However,  the  company  and  platoon  exercises  had  been  carefully  designed  so  that  the  unit  in 
training  would  get  all  the  action.  Developers  had  to  point  out  that  no  one  company  or  platoon 
within  the  battalion  would  encounter  all  of  the  activity  that  tfiey  would  during  their  own 
exercises,  even  though  the  unfolding  battle  should  be  the  same;  the  activity  would  be 
distributed  among  all  of  the  participating  companies  and  platoons. 


•  Other  Training  Priority  Guidance.  Examine  and  document  any  training- 
specific  guidance  that  you  have  been  given.  For  example,  your  guidance  may  be 
restrictive  with  respect  to  the  specific  focus  of  the  training,  such  as  emphasis  on 
combat  service  support  (CSS).  Consideration  of  where  the  program  is  intended  to 
fit  within  the  training  strategy  will  also  help  to  define  the  scope.  As  you  define 
the  range  of  training  options,  examine  and  analyze  the  choices  available  in  order 
to  help  focus  the  training  intent. 

•  Trainer  and  Other  Resources.  Know  what  training  resources  you  are  expected 
to  plan  for.  For  example,  you  may  be  able  to  plan  on  the  use  of  a  dedicated 
training  cadre  or  observer/  controller  (0/C)  team,  or  you  may  need  to  design  for  a 
“pick-up”  team  of  observers.  The  training  program  may  be  destined  for  a  single 
implementation  site,  or  you  may  be  developing  a  completely  exportable  package. 
Implementation  options  may  include  some  field  deployment  of  participants  or 
minimal  physical  realism.  Training  time  may  be  limited  to  8-hour  blocks,  or  you 
may  be  asked  to  design  for  continuous,  24-hour  operations. 

The  input  for  this  activity  may  be  external  to  your  training  development  efforts.  It  is 
possible  that  the  requirement  to  develop  training  for  a  particular  type  of  unit,  mission,  and 
simulation  will  be  specified  for  you  by  the  proponent.  In  other  cases,  the  proponent  will  rely  on 
you,  the  training  developer,  to  identify  alternatives  and  make  recommendations. 

As  you  go  on,  you  may  need  to  revisit  the  decisions  that  have  been  made.  Inconsistencies 
between  and  among  training  program  components  may  emerge  and  require  resolution.  For 
example,  you  could  discover  that  the  capabilities  of  the  selected  technology  are  not  sufficient  for 
the  intended  unit  size  or  mission  type. 

In  the  development  methodology,  this  first  activity  and  its  product  serve  as  input  to  all 
other  activities.  The  decisions  that  you  document  are  central  to  virtually  every  activity  in  the 
remainder  of  the  development  process.  They  form  the  basic  structure  for  the  exercises,  serving 
as  the  basis  for  your  development  objectives.  For  that  reason,  it  is  essential  that  you  make  sure 
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that  the  decisions  are  (1)  clearly  spelled  out,  and  (2)  available  to  and  understood  by  the 
design/development  team. 

Formative  Evaluation 

At  this  very  early  stage,  the  most  useful  formative  evaluation  will  be  conducted  in  a 
briefing  to  the  training  proponent.  Your  goal  is  to  get  concurrence  on  the  decisions  made,  so  the 
briefing  should  present  those  decisions  and  the  reasons,  as  well  as  any  outstanding  issues  to  be 
decided.  This  will  help  to  ensure  that  your  design  and  development  efforts  are  in  accordance 
with  the  intentions  of  the  proponent. 

This  phase  assumes  that  a  needs  assessment  or  front  end  analysis  has  already  been 
conducted,  and  that  it  is  the  basis  for  these  initial  constraints.  If  it  has  not,  it  is  incumbent  on  you 
to  request  (or  perform)  the  analysis.  Additionally,  now  is  the  time  to  examine  any  decisions 
made  on  the  basis  of  the  analysis,  and  identify  for  the  proponent  any  that  may  have  negative 
consequences.  Your  understanding  of  and  relationship  with  the  proponent  will  help  you  decide 
whether  you  can  challenge  the  decisions  or  the  analysis  itself. 

By  the  same  token,  ensure  that  both  the  proponent  and  the  design  and  development  team 
have  thought  about  the  implications  of  the  decisions.  The  initial  design  phase  can  be  an  exciting 
time,  with  seemingly  boundless  opportunities  for  innovative  approaches  and  ambitious  (even 
grandiose)  plans.  Try  to  anticipate  the  efforts  that  will  be  required  to  carry  out  those  plans,  and 
be  willing  to  modify  those  plans  if  they  turn  out  to  be  unrealistic  in  execution. 
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PHASE  2.  DESIGNATE  TRAINING  OBJECTIVES 


Activity  2.1 
Activity  2.2 
Activity  2.3 


Identify  task  sources,  tasks,  and  standards. 
Refine  task  list  for  simulation  support. 
Select  tasks  that  support  the  mission. 


During  this  phase,  you  will  determine  the  tasks  that  will  be  the  focus  of  the  training 
program.  The  three  activities  will  involve  finding  sources  of  information  and  extracting  the 
complete  task  list  (Activity  2.1),  refining  the  list  to  include  only  those  tasks  that  can  (or  should) 
be  trained  using  the  selected  simulation  (Activity  2.2),  and  then  further  refining  the  list  to  include 
only  those  tasks  that  also  support  the  selected  mission  type  (Activity  2.3). 

Either  of  the  two  refinement  activities  (2.2  or  2.3)  can  be  done  first.  It  makes  sense  to  do 
the  one  that  you  expect  will  result  in  the  most  drastic  cuts  first,  so  that  you  can  avoid  reviewing  a 
large  number  of  tasks  in  one  activity  that  get  set  aside  in  the  next  activity.  The  final  list, 
indicated  as  the  product  of  Activity  2.3,  should  reflect  the  results  of  both  activities,  no  matter 
which  one  you  do  first. 

The  goal  in  this  phase  is  to  develop  a  list  of  tasks  that  serves  three  purposes: 

1 .  It  provides  the  performance  structure  around  which  the  scenario  will  be  constructed. 

2.  It  is  the  foundation  of  the  observation  and  feedback  mechanism  within  the  exercise. 

3.  It  defines  the  training  objectives  for  the  intended  users. 


Activity  2.1  Identify  task  sources,  tasks,  and  standards. 
ProductfOutcome;  Tasks  and  standards;  task  sources. 

Formative  Evaluation;  Internal  and  proponent  review.  Training  and  doctrine  agency 
review  after  tasks  are  selected  (both  Activities  2.2  and  2.3). 


The  first  activity  in  Phase  2  involves  identifying  task  sources  for  the  unit  type  and 
echelon,  and  then  preparing  a  complete  list  of  all  of  the  tasks. 

For  Army  training  projects,  the  most  obvious  task  sources  are  the  Army  Training  and 
Evaluation  Program  (ARTEP),  Mission  Training  Plans  (MTPs)  and  field  manuals  (FMs).  Other 
sources  include  training  materials  prepared  by  the  proponent  agencies  and  task  lists  developed  by 
the  proponent  schools.  Consider  and  examine  all  sources  that  are  known  or  suggested  to  you. 
You  might  not  use  them  all — they  may  be  redundant  or  in  an  early  development  stage  or  out  of 
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date,  for  example.  You  should  keep  a  record  of  all  sources  considered  and  the  reason  you  did  not 
use  particular  sources  that  you  considered  (in  case  there's  a  question  later). 

Make  sure  that  you  have  the  most  recent  approved  versions.  If  you  know  that  another 
version  is  coming  out  soon,  you  should  review  the  draft  if  you  can,  see  what  differences  there 
will  be,  and  then  get  a  decision  as  to  whether  to  use  the  current  or  the  emerging  version.  Your 
decision  will  depend  on  when  the  change  or  update  will  be  released  and  how  vast  the  differences 
are.  Using  a  different  version  during  development  than  the  training  audience  uses  can  cause 
confusion  when  the  program  is  implemented. 

For  a  source  to  be  useful,  it  must  contain  tasks  for  the  mission  type  and  for  the  unit  type 
and  echelon  (the  METT-T  specified  during  Phase  1).  The  tasks  must  clearly  describe  how  to 
perform,  under  what  conditions,  and  to  what  standard.  Sometimes,  information  from  several 
sources  can  be  combined  to  yield  complete  task  analytic  information.  But  if  you  are  using 
multiple  sources,  check  your  final  list  of  tasks  for  redundancies  and  cases  in  which  certain  tasks 
are  actually  subtasks  of  others  (a  more  subtle  form  of  redundancy). 

As  you  read  the  task  analytic  materials,  you  will  need  to  identify: 

•  Tasks — lowest  level  of  collective  behavior  that  has  accompanying  conditions  and 
standards.  (In  some  sources,  these  may  be  called  subtasks  or  task  steps.) 

•  Conditions — description  of  situation,  environment,  and  initiating  cues  that  should 
cause  a  task  to  be  performed. 

•  Standards — statement  of  correet  accomplishment  of  a  task. 

You  will  usually  be  expected  to  use  tasks  that  can  be  documented  in  official  sources  (e.g., 
ARTEP-MTPs).  Because  your  objective  is  to  design  training  that  uses  task  analytic  information 
accepted  by  the  organization,  it  makes  sense  to  use  existing  documentation  when  possible.  A 
multiplicity  of  task  lists  that  are  only  different  because  of  personal  preferences  only  serves  to 
confuse  training  strategies. 


I  EXAMPLE:  Discovering  task  sources  (or,  the  exception  that  establishes  the  rule) 

I  GIVEN:  A  requirement  to  design  training  for  brigade  staffs  that  stresses  Synchronization  and 
i  integration. 

I  In  this  project,  the  tasks  were  not  limited  to  Official  sources.  Instead,  they  were  derived  from  a 
j  controlled  job  analysis  process  involving  performance  of  segments  of  the  exercise,  roleplay, 

I  diary-keeping,  and  documentation. 


The  product  for  Activity  2.1  is  a  fairly  comprehensive  list  of  tasks  that  describe  the 
expected  behavior  of  the  training  audience,  as  found  in  the  source  documents  or  derived  by  some 
other  proponent-accepted  process.  In  Activity  2.2,  the  list  will  be  refined  in  order  to  determine 
which  tasks  can  be  represented  on  the  simulation. 
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Your  work  in  preparing  the  task  list  is  not  likely  to  cause  revisions  to  the  initial  decisions 
unless  the  quality  of  task  analytic  information  is  so  poor  that  structured  training  for  the  selected 
mission  and/or  unit  cannot  be  developed  until  the  quality  of  the  task  analysis  is  improved.  It  is 
also  unlikely  that  you  will  have  to  revise  this  task  list  later,  unless  other  sources  of  task 
information  are  discovered,  or  doctrine  or  proponent  guidance  changes  during  development. 

Formative  Evaluation 

As  you  complete  the  list  of  tasks,  the  formative  evaluation  should  be  an  internal 
process — ^you  are  reviewing  and  updating  your  own  work.  Once  refinements  to  the  task  list  are 
completed  in  Activities  2.2  and  2.3,  you  should  request  a  proponent  review  and  a  review  by  the 
appropriate  doctrine  and  training  agencies. 


Activity  2.2  Refine  task  list  for  simulation  support. 

Product/Outline;  Task  list  annotated  to  show  tasks  that  can  be  fully  or  partially  performed 
and  observed  in  the  simulation. 

Formative  Evaluation;  Internal  and  proponent  review.  Training  and  doctrine  agency 
review  after  tasks  are  screened  for  simulation  support  (this  activity)  and  mission  support 
(Activity  2.3). 


This  activity  requires  you  to  make  judgments  about  which  tasks  can  be  included  in  the 
training,  based  on  simulation  capabilities.  You  may  find  many  tasks  that  you  want  to  train,  but 
that  cannot  (or  should  not)  be  trained  using  the  selected  simulator  or  simulation.  Some  tasks  can 
be  partially  represented,  and  you  will  have  to  judge  whether  those  parts  are  critical  and  are  well 
enough  represented  to  justify  their  inclusion  in  the  training. 

The  refining  process  uses  the  task  list  from  Activity  2.1.  If  you  have  already  done  the 
mission  screen  on  that  task  list  (Activity  2.3),  then  the  resulting  reduced  list  will  serve  as  your 
starting  point.  In  order  to  make  the  judgments  of  whether  or  not  each  task  should  be  included, 
based  on  simulation  capability,  you  need  a  rule-based  system  that  specifies  the  basis  for  the 
decisions.  It  is  not  sufficient  for  you  to  simply  look  at  tasks  or  subtasks  and  decide  whether  or 
not  you  feel  that  they  can  be  represented  on  the  simulator/simulation.  And  you  should  have  at 
least  two  people  independently  make  ratings  of  task  suitability.  Afterwards  they  can  work 
together  to  arrive  at  a  consensus. 
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EXAMPLE:  Screening  tasks 

GIVEN:  A  requirement  to  design  SIMNET-based  training  for  platoons 

Developers  modified  an  approach  developed  by  Burnside  (1990)  to  screen  ARTEP-MTP  tasks 
for  training  on  SIMNET.  To  use  that  approach;  suitability  Judgments  are  first  made  at  the  lowest 
level  of  task  detailing  (i.e.,  subtask  standards)  and  then  the  ratings  are  aggregated  to  higher 
levels  (subtask  or  task).  Developers  then  follow  a  set  of  rules  to  determine  task  traihability.  In 
this  effort,  developers  aggregated  ratings  of  subtask  sfandarc/s  only  to  the  Subfask  level,  rather 
than  to  the  task  level,  this  permitted  them  to  make  judgments  of  trainability  at  the  subtask  level 
and  thus  cover  parts  of  tasks  in  the  exercises. 


Keep  in  mind  that  a  great  deal  of  live  simulation  can  go  on  around  a  constructive  or 
virtual  simulation  exercise.  If  a  task  isn’t,  strictly  speaking,  supported  by  the  simulation,  but  it 
can  be  performed,  observed,  and  evaluated  “live,”  then  it  may  be  a  good  candidate  for  inclusion. 
The  simulation-based  training  can  provide  a  rich  environment  for  performance  of  tasks  that  are 
not  themselves  simulation-supported. 

As  you  work  through  this  activity,  you  will  have  to  be,  or  have  access  to,  someone  who  is 
very  familiar  with  the  capabilities  of  the  simulator/simulation.  It  will  not  be  enough  to  look  at 
the  capabilities  descriptions  provided  by  the  vendor,  nor  to  take  the  word  of  the  development 
engineers  and  technicians.  There  is  no  adequate  substitute  for  first-hand  knowledge  of  how  the 
simulation  works.  Unless  you  yourself  are  expert  on  the  simulation,  you  will  need  to  work 
closely  with  a  technician  who  can  demonstrate  the  functions  that  you  need  to  replicate. 

The  work  on  this  activity  should  not  require  you  to  revise  earlier  work  (i.e..  Activities  1.1 
and  2.1,  and  possibly  2.3)  unless  it  turns  out  that  no  tasks  survive  the  cut  and  the  technology  is 
totally  inappropriate  to  the  imit  type  or  mission.  The  decisions  made  in  this  activity  on  the  basis 
of  simulation  capabilities  should  only  change  in  reaction  to  upgrades  to  the  simulation 
technology  or  further  information  about  capabilities  (i.e.,  experience  that  shows  that  the 
capabilities  are  not  what  you  thought  they  were). 

Formative  Evaluation 

The  formative  evaluation  requirement  still  includes  reviews  performed  by  you  and  other 
developers  on  the  project.  When  both  Activity  2.2  and  Activity  2.3  are  completed,  you  will  set 
up  a  more  formal  review  by  others  who  are  familiar  with  the  simulation,  the  tasks,  and  the  needs 
assessment. 
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Activity  2.3  Select  tasks  that  support  mission. 

Product/Outcome;  Reduced  task  list,  annotated  to  show  the  tasks  (or  parts  of  tasks)  that 
should  be  performed  in  the  context  of  the  mission,  along  with  appropriate  conditions  and 
standards. 

Formative  Evaluation:  Internal  and  proponent  review.  Training  and  doctrine  agency 
review  after  tasks  are  screened  for  simulation  support  (Activity  2.2)  and  mission  support 
(this  activity). 


At  this  point,  you  are  ready  to  decide  which  tasks  the  training  should  include,  based  on 
whether  they  are  appropriate  to  the  mission  selected.  Tasks  and  standards  that  are  on  the  list  that 
you  prepared  in  Activity  2.1  (and  that  passed  the  simulation  screen  in  Activity  2.2  if  that  has 
already  been  done),  should  now  be  examined  in  the  context  of  the  mission  type.  The  product  of 
this  activity  is  the  list  of  training  objectives  that  support  the  selected  mission. 

In  addition  to  the  type  of  mission,  there  may  also  be  other  mission-related  considerations. 
For  example,  maybe  you  have  been  told  that  exercises  need  to  incorporate  close  air  support,  or 
extensive  use  of  indirect  fire,  or  should  not  incorporate  any  static  defense.  Or  you  may  be  given 
a  general  focus,  such  as  NTC  training  needs  and  deficiencies,  or  the  commander's  guidance  or 
mission-essential  task  list  (METL)  emphasis.  You  should  tailor  the  task  list  according  to  any  of 
those  "other"  constraints  or  guidelines  imposed  on  you  by  the  proponent,  to  ensure  that  the 
appropriate  tasks  are  included. 

The  list  of  tasks  and  standards,  screened  for  simulation  and  mission  support,  now 
represents  the  training  objectives.  This  list  is  used  in  the  activities  in  Phase  3,  where  you  will 
construct  the  tactical  scenario  that  will  require  units  to  perform  the  tasks.  As  you  develop  the 
scenario,  you  may  find  that  not  all  of  the  selected  tasks  can  included  in  a  single  scenario.  If  you 
decide  later  that  you  will  not  train  a  task  after  all,  make  sure  you  revise  the  task  list  and  make  a 
note  of  the  reason.  For  your  own  reference  and  for  future  development  work,  it  is  important  to 
maintain  an  audit  trail  on  why  specific  tasks  were  or  were  not  selected. 

Formative  Evaluation 

Once  the  list  is  finalized  (or  temporarily  finalized),  and  you  have  done  another  internal 
review,  you  will  want  to  get  proponent  concurrence  on  what  will  be  trained.  A  briefing  will  let 
you  describe  the  procedure  you  followed,  explain  and  defend  your  decisions,  and  get  approval  to 
proceed  with  development.  You  should  present  complete  task  descriptions,  and  be  prepared  to 
show  the  tasks  that  were  rejected  and  why  (e.g.,  didn’t  support  the  mission  or  not  supported  by 
the  simulation).  You  should  also  try  to  involve  any  interested  training  and  doctrine  agencies,  so 
that  their  eeirly  concurrence  is  obtained. 
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PHASE  3.  DESIGN  SCENARIO  AND  EXERCISE  STRUCTURE 


Activity  3.1 
Activity  3.2 
Activity  3.3 


Design  the  scenario. 

Prepare  exercise  context  and  specifications. 
Outline  events  and  build  exercise. 


During  this  phase,  which  mirrors  the  Design  phase  of  the  Systems  Approach  to  Training 
(SAT),  you  will  plan  and  outline  the  tactical  scenario  that  will  be  the  context  for  the  exercises, 
and  make  decisions  about  the  exercise  structure  for  the  training.  This  is  a  lengthy  phase, 
comprising  three  activities. 

Throughout  the  three  activities,  give  some  thought  to  task  sequencing.  Even  though  the 
unit  will  be  performing  tasks  in  whatever  sequence  the  scenario  requires,  you  may  have  some 
flexibility  in  designing  the  scenario  so  that  tasks  will  occur  in  an  order  that  you  control.  The 
most  likely  sequences  that  make  sense  from  a  training/learning  standpoint  are  crawl-walk-run, 
natural  order,  hierarchical  order,  and  easy-to-difficult.  The  intent  of  each  of  these  is  as  follows: 

•  Crawl-walk-run — Tasks  are  repeated  several  times  (technically,  three  times)  under 
increasingly  demanding  conditions,  in  order  to  hone  the  unit's  performance  skills. 
Conditions  are  usually  made  more  demanding  by  changing  the  mission,  increasing  the 
enemy  strength  or  capability,  or  requiring  the  unit  to  perform  in  more  challenging 
terrain,  less  time,  or  lower  readiness  levels  (the  five  METT-T  elements).  This 
sequencing  provides  opportunities  to  reinforce  training  and  to  work  toward 
automation  in  performance  (i.e.,  making  performance  more  automatic,  as  is  the  goal 
with  contact  and  battle  drills). 

•  Natural  order — This  sequencing  anticipates  performance  of  tasks  in  the  normal  order  in 
which  they  are  performed.  This  sequencing  will  generally  happen  without  explicit 
effort  because  of  the  tactical  context  in  which  the  exercises  are  embedded.  Just  as  the 
events  follow  a  natural  chronological  order,  because  they  occur  within  the  framework 
of  the  start-to-finish  mission,  the  tasks  for  a  given  unit  will  be  required  in  a 
chronological  order.  This  does  not  imply  that  this  is  the  best  order  for  learning  tasks, 
but  it  does  provide  a  structure  for  performance  cueing  that  reflects  real  world 
requirements. 

•  Hierarchical  order— Tasks  are  examined  to  determine  whether  any  of  those  selected 
actually  contribute  to,  or  are  subtasks  of,  other  tasks.  For  example,  being  able  to 
execute  a  line  formation  is  required  in  order  to  execute  an  action  drill.  The  mission 
scenario  itself  would  be  designed  to  permit  early  focus  on  lower  level  tasks  in  the 
hierarchy,  and  later  emphasis  on  the  higher  level  tasks.  In  order  to  make  this  happen, 
however,  you  need  to  determine  what  the  hierarchical  relationships  are. 
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•  Easy-to-difficult— Tasks  are  rank-ordered  according  to  difficulty,  and  the  exercise  or 
segments  are  designed  so  that  easy  tasks  are  required  first,  and  more  difficult  tasks  are 
gradually  introduced.  The  advantage  provided  is  that  units  have  time  to  adjust  to  the 
training  environment  before  they  are  required  to  perform  the  more  difficult  tasks.  The 
disadvantage  is  that  you  need  to  obtain  ratings  of  task  difficulty  somewhere,  either  from 
subject  matter  experts  or  from  other  research.  There  is  little  evidence  to  suggest  that 
learning  occurs  better  using  this  sequencing,  unless  it  is  incidentally  supporting  one  of 
the  other  three  sequencing  schemes  above. 

It  is  unlikely  that  you  will  have  a  stated  directive  for  sequencing.  All  of  the  sequencing 
principles  described  above  are  appealing,  and  even  seem  to  be  redundant  in  some  cases.  You 
may  decide  to  work  with  just  one  approach,  or  to  use  different  approaches  for  particular  tasks  and 
parts  of  the  training. 

This  process  of  structuring  the  scenario  also  serves  to  clarify  the  task  standards,  from  the 
general  levels  usually  found  in  ARTEP-MTPs  to  very  specific  METT-T-driven  standards.  For 
each  of  these  structuring  considerations  (i.e.,  incorporating  realism  at  higher  levels,  using 
documented  enemy  tactics,  using  selected  terrain,  maintaining  the  flow,  and  sequencing)  you 
should  consult  the  proponent  before  continuing  with  this  phase  of  development. 

With  more  complex  scenario  situations,  it  will  not  be  a  simple  matter  to  go  beyond 
natural  order  in  sequencing  of  particular  tasks.  However,  whole  segments  of  scenarios, 
encompassing  a  large  number  of  tasks,  may  be  structured  so  as  to  increase  the  difficulty  of 
conditions  or  to  take  advantage  of  multiple  task  hierarchies. 


Activity  3.1  Design  the  scenario 

Product/Outline;  Draft  of  the  "concept  of  the  operation"  with  sketch  of  graphic  overlay, 
draft  of  unit  operations  order  (OPORD),  or  other  representation;  for  training  unit  and  higher 
echelons  as  necessary. 

Formative  Evaluation;  Map  exercises 


If  you  have  been  using  the  methodology  as  presented  up  to  this  point,  you  now  have  the 
selected  task  list,  showing  which  tasks  can  and  should  be  trained  in  the  context  of  the  mission 
scenario  that  will  be  designed.  You  also  know  about  any  other  constraints  on  the  design,  such  as 
the  amount  of  time  available,  the  need  for  stand-alone  exercise  partitions,  sequencing 
preferences,  the  nature  of  the  enemy,  and  the  general  area  (terrain)  where  the  exercise  will  be 
conducted.  You  have  probably  already  begun  to  form  the  tasks,  unit,  enemy,  and  terrain  into  an 
exercise  or  series  of  exercises. 
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Because  structured  training  requires  that  the  exercise  should  be  presented  in  the  context 
of  a  credible  situation,  you  will  be  concerned  with  realism  in  all  aspects  of  the  exercise.  For 
battle-oriented  exercises,  realism  requires  you  to  develop  the  mission  scenario  for  the  training 
unit  level  and  for  one  or  two  echelons  up  (e.g.,  platoon-level  exercises  require  supporting 
mission  outlines  for  company-  and  battalion-levels).  The  higher  echelon  missions  are  developed 
because  (1)  the  process  ensures  that  the  imit’s  mission  is  sound,  and  (2)  the  higher-order 
information  is  needed  as  you  prepare  scripted  messages  and  other  inputs  to  cue  the  unit’s 
performance.  Mission  scenarios  will  generally  be  presented  in  the  form  of  operations  orders 
(OPORDs). 

For  exercises  that  focus  on  skills  that  are  less  directly  battle-oriented  (e.g.,  decision¬ 
making,  planning),  the  realism  will  be  ensured  by  means  of  various  tactical  products  or  messages 
that  cue  the  performance  of  the  selected  tasks.  While  these  will  be  based  on  a  well-thought-out 
mission,  you  may  not  need  to  develop  the  complete  OPORD. 

Similarly,  the  enemy  (if  there  is  one)  should  be  realistic.  You  specified  the  enemy  type  in 
Phase  1,  and  you  need  to  follow  through  on  that  decision  in  prescribing  the  enemy  capabilities 
and  tactics  in  the  exercises.  That  is,  the  enemy  needs  to  be  configured  and  to  behave  in  the 
exercises  according  to  some  model.  Make  sure  that  the  enemy  to  be  presented  is  doctrinally 
appropriate  for  the  size,  composition,  and  mission  of  the  unit  being  trained. 

In  order  to  maintain  the  realism  of  a  battle-oriented  exercise  throughout  its  execution,  you 
should  develop  the  entire  scenario  for  a  designated  piece  of  terrain,  and  make  the  exercise  move 
across  that  terrain  in  a  real-time,  real-space  fashion.  In  this  approach,  the  entire  battle  or  series  of 
battle  events  will  occur  in  the  exercises  as  a  continuous  flow  on  a  (large)  selected  piece  of 
ground.  If  different  pieces  of  the  exercise  occur  in  separated  locations  or  times,  it  may  require 
additional  narratives  explaining  how  the  unit  got  where  they  are,  which  could  result  in  breaks  in 
the  flow  of  the  battle  and  other  awkward  side-effects. 

However,  if  you  find  that  the  missions  or  operations  require  extensive  terrain  or  long 
periods  of  time  in  order  to  incorporate  all  of  the  tasks,  and  that  there  are  long  periods  of  non¬ 
training  time  (i.e.,  road  marches  may  not  be  part  of  the  training  objective  list),  then  one  solution 
is  to  deliberately  break  up  the  mission  into  smaller  partitions  of  the  mission.  Each  partition  then 
focuses  on  specific  tasks,  occurs  on  specified  terrain  during  a  specified  time  period,  and  may  be 
separated  from  the  other  parts  in  time,  space,  or  other  conditions. 

Finally,  realism  extends  to  other  “friendly”  personnel  or  units.  In  battle-oriented 
exercises,  you  will  need  to  generate  representations  of  higher  echelon  units  and  supporting, 
subordinate,  and  adjacent  units.  Similar  representations  will  often  be  required  in  other  types  of 
exercises  as  well,  to  provide  the  cues  that  drive  performance. 

The  requirement  in  Activity  3.1  is  to  prepare  an  outline  or  draft  of  the  training  unit's 
mission.  This  product  could  look  like  a  concept  of  the  operation,  or  a  course  of  action,  or  a 
sketch  of  a  map  with  notes  to  indicate  what  will  happen  where.  For  some  exercises,  the  actual 
ground  where  the  battle  is  fought  is  important,  but  the  selected  tasks  are  not  performed  on  that 
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ground.  For  example,  in  battalion  staff  training,  the  staff  generally  stays  at  command  post 
locations  throughout  the  battle,  rather  than  being  out  on  the  battlefield.  As  a  result,  you  may  also 
need  to  prepare  a  matrix  to  indicate  what  events  should  happen,  and  what  the  enemy,  the 
controlled  (notional)  vmits,  and  the  training  unit  should  be  doing. 

Your  product  needs  to  indicate  the  initial  unit  and  OPFOR  locations,  major  events, 
participants  (friendly  and  enemy),  and  the  area  of  terrain  that  will  be  used.  As  you  do  this,  keep 
in  mind  that  you  may  also  need  to  develop  the  supporting  mission  for  one  or  two  organizational 
levels  (or  echelons)  up,  so  the  final  training  scenario  for  the  training  unit  needs  to  have  realistic 
counterparts  at  the  higher  levels.  Within  the  product,  make  some  notation  of  when  and/or  where 
the  selected  tasks  (from  Phase  2)  are  expected  to  be  performed  in  the  course  of  the  execution. 

For  a  battle-oriented  exercise,  the  products  may  include  any  or  all  of  the  following: 

•  description  of  the  mission,  commander’s  intent,  and  concepts  for  any  combat  support 
(CS)  and  CSS  assets  that  are  included  in  the  exercise; 

•  situational  template  that  shows  your  plan  for  the  enemy  (i.e.,  location,  objective, 
mission,  intent,  and  course  of  action); 

•  plan  for  the  terrain  that  will  support  the  mission  (area  of  operation);  a  rough  sketch  of 
the  graphic  overlay  of  control  measures  for  the  mission;  and 

•  scenario  event  timeline  showing  the  activities  of  both  friendly  and  enemy  forces. 

This  activity  and  the  next  one  (i.e..  Activities  3.1  and  3.2)  are  closely  tied  and 
interdependent.  You  should  check  frequently  on  the  correspondence  among  the  training  unit's 
activities  or  mission,  the  higher  levels'  activities  or  mission,  and  the  detailed  scenario  outline 
(Activity  3.2),  in  order  to  make  sure  that  they  are  tactically  correct  and  fully  coordinated. 

The  work  on  this  activity  (in  conjunction  with  the  next  two  activities)  might  result  in 
changes  in  products  of  earlier  activities.  For  instance,  you  may  find  that  this  product  does  not 
trigger  performance  of  all  of  the  selected  tasks  and  that  you  are  unable  to  adjust  it  so  all  tasks  can 
occur  in  a  tactically  realistic  scenario.  In  that  case,  you  need  to  go  back  to  Activity  2.3  and 
correct  the  selected  task  list  so  that  it  is  accurate  with  respect  to  tasks  that  are  going  to  be  trained. 

Formative  Evaluation 

In  order  to  check  on  whether  the  product  supports  all  of  the  selected  tasks,  and  to  verify 
that  the  mission  is  tactically  adequate,  you  should  conduct  a  map  exercise  of  the  mission.  This 
will  ensure  that  the  terrain  selected  is  appropriate  and  that  the  operational  concept  is  rational  and 
intuitively  acceptable.  Use  content  experts  in  this  process,  so  that  you  get  other  views  of  whether 
the  mission  makes  good  tactical  sense. 

As  with  any  map  exercise,  you  will  be  examining  the  reasoning  behind  the  tactical  plan. 
Specifically,  you  will  be  examining  every  decision  made  concerning  the  unit's  mission,  the 
higher-order  mission,  and  the  intended  enemy  behavior.  You  and  the  other  reviewers  must 
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rigorously  scrutinize  and  challenge  each  decision  at  this  point,  in  order  to  avoid  rewriting  and 
changing  things  later  in  development  when  the  products  are  more  intricately  interdependent. 


Activity  3.2  Prepare  exercise  context  and  specifications. 

Product/Outcome:  Context,  specifications,  and  execution  details  for  exercises. 
Formative  Evaluation:  Simulation-controlled  exercise. 


In  this  activity,  you  will  prepare  the  context  statements,  initial  unit  specifications,  and 
execution  descriptions  for  the  exercise  or  exercise  partitions. 

If  you  are  going  to  partition  the  mission  within  the  exercise,  you  should  consider  two 
things:  logical  stopping  points  within  the  exercise  (e.g.,  initial  enemy  recoimaissance  elements, 
trigger  points,  phase  lines,  or  mission  transition  points  such  as  a  battle  handover  to  defeat  enemy 
attack),  and  the  desired  length  of  each  segment.  Look  for  breakpoints  that  are  plus  or  minus  15 
minutes  of  what  you  really  want;  estimates  at  this  point  are  usually  not  very  aecurate,  so  be 
flexible.  If  it  turns  out  that  some  of  the  segments  are  too  long  or  short,  you  may  shift  the 
breakpoints  or  move  performance  cues  and  events  from  one  segment  to  another. 

Even  if  you  are  not  partitioning  the  exercise  for  purposes  of  providing  alternative  entry 
points,  be  cautious  of  allowing  a  long  period  of  time  to  go  by  with  no  after  action  review  (AAR). 
There  will  be  too  great  a  chance  that  proficient  performance  will  not  be  reinforced,  that 
nonproductive  behaviors  will  not  get  corrected,  and  that  important  learning  points  will  be 
forgotten  by  the  time  the  AAR  occurs. 

Using  the  task  list  (from  Phase  2)  and  the  draft  scenario  (from  Activity  3.1)  as  input,  draft 
a  narrative  or  graphic  description  of  the  exereise  (or  of  each  partition  of  the  exercise),  including 
starting  and  approximate  ending  locations,  the  events  that  will  occur,  and  the  tasks  that  will  be 
performed  by  the  unit.  If  you  are  working  with  partitions,  check  to  be  sure  that  they  are  of  the 
appropriate  length  (estimated)  and  that  they  reflect  the  sort  of  sequencing  that  you  want.  The 
time  length  that  you're  aiming  for  was  decided  earlier,  in  Phase  1 . 

You  will  probably  begin  to  formulate  the  intended  cues  that  trigger  task  performance  as 
you  go  through  this  process.  Activity  3.3  will  direct  your  attention  to  the  speeification  of  cues, 
whether  they  are  provided  by  the  simulation,  by  means  of  scripted  messages  or  contrived  events 
(higher  command  or  OPFOR),  or  by  other  participants.  Make  a  note  of  them  as  you  go  along, 
because  you  will  certainly  need  them  later. 
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In  addition  to  the  narrative  and/or  graphic  representation  of  the  exercise,  there  are  five 
kinds  of  information  that  you  should  document  for  battle-oriented  exercises: 

1 .  The  friendly  and  enemy  situations. 

•  Describe  the  friendly  situation  for  the  training  unit  and  for  any  higher, 
subordinate,  or  adjacent  units  whose  actions  will  cue  the  training  unit. 

•  The  enemy  situation  should  include  the  enemy  capabilities,  organization,  scheme 
of  maneuver,  and  plan  or  intent.  The  descriptions  should  be  enough  to  generate 
the  appropriate  intelligence  reads  that  will  cue  the  training  unit’s  intelligence  staff. 

•  Prepare  a  short  description  of  the  immediately  preceding  events  that  brought  the 
forces  to  this  point.  If  the  exercise  is  partitioned,  this  will  be  a  description  of  what 
should  have  happened  in  the  one  or  two  immediately  previous  segments;  if  you 
aren't  partitioning  it  will  be  what  happened  immediately  before  the  exercise 
started. 

•  Start  now  to  think  about  how  you  will  bring  the  unit’s  knowledge  of  the  enemy  up 
to  a  realistic  level.  In  the  next  phase,  you  will  prepare  the  intelligence  summaries, 
estimates,  and  narratives  that  are  the  key  documents  for  telling  the  unit  what  they 
would  know  at  the  exercise’s  start  point. 

•  Specify  the  starting  points  for  all  of  the  represented  units  in  terms  of  approximate 
location  (4-digit  grid  or  control  measure). 

2.  Unit  specifications.  You  will  need  certain  information  for  all  of  the  represented  units, 
vehicles,  weapons  systems,  and  so  on. 

•  Identify  every  friendly  or  enemy  unit  that  needs  to  be  represented  in  the 
simulation.  Give  each  of  those  units  a  unit  identifier,  tell  how  they're  organized, 
and  tell  how  they'll  be  represented  (e.g.,  voice,  simulated,  notional,  or  scripted). 

•  For  any  virtual  simulation  systems  or  vehicles  that  will  be  operated  by  humans, 
give  complete  initial  specification  (e.g.,  system  number  and  type,  call  letters, 
location/azimuth/formation,  maintenance/fliel/ammo  status). 

•  Develop  the  specifications  for  each  automated  or  semi-automated  system  (e.g., 
unit  type,  system  number  and  type,  friendly/enemy,  location/azimuth/formation, 
capability).  Even  if  you  are  an  expert  on  the  simulation  technology,  you  need  to 
have  another  expert  review  your  products  to  make  sure  that  you  have  specified 
values  for  all  parameters  or  used  the  appropriate  defaults. 

3.  The  ending  point.  For  the  exercise  or  for  each  partition,  decide  on  the  ending,  in 
terms  of  the  approximate  location  and  the  event  or  condition  that  signals  the  end 
point.  In  most  cases,  because  the  exercises  are  in  scenarios,  achievement  of  a  task 
standard  will  not  be  the  indication  to  end  a  segment.  The  scenario  (i.e.,  segment  of 
the  mission)  will  usually  continue  to  a  logical  ending. 
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4.  Scenario  description.  This  should  be  more  than  just  a  mission  title  (e.g.  movement  to 
contact),  but  less  than  a  list  of  tasks  and  performance  objectives.  The  intent  will 
usually  give  a  brief  description  of  the  exercise  conditions  (e.g.,  heavy  indirect  fire, 
light  enemy  contact,  severe  time  constraints)  as  well  as  a  statement  of  what  the  unit 
does  in  terms  of  what  you  want  them  to  learn.  It  is  used  to  introduce  the  exercise  to 
potential  users. 

EXAMPLE:  Describing  the  scenario 

GiVEN:  A  brigade  staff  vignette  oh  planning  Nuclear,  Biological  and  Chemical  (NBC)  defensive 
measures  ‘ 

The  scenario  description  was  to  be  included  in  the  training  materials,  to  assist  training  planners 
in  understanding  the  vignette  scope.  The  description  read  as  followsf  ^  ^  ^  ^ 

“This  exercise  takes  place  while  the  brigade  is  planning  an  area  defense.  The 
commander  has  provided  a  friendly  course  of  action  (COA)  for  the  staff  to 
synchronize.  The  staff  has  partially  completed  missiop  analysis,  and^^t  S3  has 
decided  to  further  analyze  the  mission  in  the  area  of  NBC  defense  before  continuing 
with  COA  analysis/wargaming.  The  staff  will  identify  and  analyze  the  probable  NBC 
threat  to  support  the  enemy  COA  and  will  identify  NBC  defense  measures  to  counter 
the  threat.” 


5.  Tasks/training  objectives.  Finally,  attach  a  listing  of  the  tasks  (Phase  2)  that  are  the 
focus  of  the  exercise  of  each  partition  of  the  exercise.  This  will  help  to  demonstrate 
to  users  and  proponents  that  the  exercises  do,  in  fact,  provide  task  coverage  and  focus. 

Formative  Evaluation 

You  should  conduct  another  quality  check  on  the  feasibility  of  the  details  and 
specifications  that  you  selected  in  Activity  3.3.  This  time,  you  should  go  to  the  simulation  to  try 
out  parts  of  the  exercise.  Use  whatever  automated  or  semi-automated  features  are  available;  the 
goal  is  to  try  out  the  exercise  requirements  in  simulation,  while  including  human  players  as  little 
as  possible  (so  as  to  reduce  human  error  and  variability).  The  purposes  of  the  tryout  are  to: 

•  verify  that  everything  can  be  represented  on  the  technology; 

•  verify  that  the  exercise  works  on  the  terrain;  and 

•  ensure  that  all  capabilities,  friendly  and  enemy,  are  set  appropriately. 

In  order  to  do  this  formative  evaluation,  you  will  need  to  build  the  preliminary  files  for 
the  simulation.  Although  it  will  depend  on  the  technology  you  use,  the  simulation  files  may 
include  starting  conditions  (the  specifications  listed  in  Activity  3.2)  and  may  also  include  routes, 
graphics,  decision  points,  and  so  on. 

You  need  to  do  these  preliminary  tryouts  using  the  simulation  before  the  tryouts  with 
soldiers  (Phase  4). 
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Activity  3.3  Outline  events  and  build  exercise. 

Product/Outcome:  Master  event  list  showing  cues,  expected  performance,  and  critical 
tasks  or  subtasks;  simulation  files. 

Formative  Evaluation:  Pilot  with  knowledgeable  personnel. 


This  activity  will  lead  you  through  the  requirements  for  specifying  all  of  what  goes  on 
during  an  exercise.  The  events  within  the  exercise  must  be  carefully  controlled  so  that  they  will 
occur  when  required  in  order  to  provide  the  conditions  for  training.  "Events"  are  defined  in 
terms  of  the  cues  or  trigger  points  that  cause  the  unit,  enemy,  or  other  entity  to  take  action,  and 
the  expected  response.  Note  that  it  is  entirely  possible  for  the  unit's  reaction  to  trigger  the  next 
event  by  causing  the  enemy  or  higher  level  to  do  something,  which,  in  turn,  causes  the  unit  to  do 
something  else.  For  each  event  in  which  the  unit  acts,  the  specific  training  objectives  (i.e., 
critical  subtasks  and  standards)  should  be  designated. 

The  input  for  the  activity  is  the  draft  scenario  description  (Activity  3.1),  as  well  as 
specific  information  about  the  conditions  for  the  exercise  (Activity  3.2),  such  as  starting  and 
ending  points  and  the  initial  configurations  of  the  unit  and  enemy.  The  purpose  of  this  activity  is 
to  add  content  to  that  initial  outline.  You  will  be  producing  the  master  event  list  (which  serves  as 
the  storyboard  for  the  structured  training  exercise),  and  refining  the  simulation  files  based  on  that 
storyboard. 

For  each  event  in  the  exercise  or  segment,  your  master  event  list  should  include  decisions 
on: 

•  what  will  start  the  event  (e.g.,  an  order  to  move  out,  a  flight  of  helicopters,  unit  reaches 
a  particular  control  point);  and  how  it  will  be  provided  (e.g.,  a  scripted  message  from 
the  higher  headquarters  roleplayers); 

•  what  the  unit's  response  should  be  (e.g.,  move  out  in  column,  active  air  defense,  report); 
and 

•  what  critical  task  you  want  to  observe,  and  what  it  should  look  like. 

Note  that  the  unit's  reaction  to  the  starting  cue  could  be  the  cue  for  another  action  that 
might  be  the  cue  for  the  next  event.  It  might  also  be  the  cue  for  some  enemy  action  (e.g.,  when 
unit  crosses  the  line  of  departure  [LD],  send  enemy  patrol  on  the  route  from  NK2 15885). 

The  longer  the  exercise  (especially  if  you  don't  have  partitions),  the  more  different 
directions  it  can  take,  and  the  harder  it  becomes  to  make  a  definitive  events  guide.  But  the  intent 
of  structured  training  is  to  provide  the  conditions  for  the  unit  to  learn  specific  things  in  a  rational 
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(not  random)  order.  Repeated  practice  under  standardized  conditions  limits  performance 
randomness  and  makes  it  possible  to  identify  changes  in  performance  and  lingering  performance 
deficiencies.  These  judgments  of  improvement  or  deficiency  are  made  by  reference  to  the  task 
standards.  It  is  important,  then,  to  work  at  making  things  happen  your  way,  rather  than  throwing 
the  unit  in  and  letting  things  happen  as  they  will.  The  Phase  4  trials  will  help  you  capture  the 
likely  unit  actions  and  verify  that  you  have  anticipated  and  represented  the  most  probable  flow  of 
the  mission. 

When  you  select  the  critical  tasks  for  an  event,  be  conservative.  Try  not  to  recommend 
observation  of  more  than  about  four  or  five  tasks  at  a  time.  Experience  in  several  projects 
suggests  that  this  is  about  the  limit  for  careful  observation.  These  are  the  tasks  or  behaviors  that 
you  are  directing  the  exercise  observer  to  evaluate  and  discuss  in  the  AAR.  Be  selective,  and  pay 
attention  to  the  sequencing.  For  example,  after  you  have  called  for  observation  of  a  platoon 
executing  a  tine  formation  in  two  events,  you  might  direct  observers  to  watch  for  something  else, 
something  more  difficult,  in  the  next  contact  drill. 

It  is  recommended  that  you  call  for  observation  of  a  task  in  more  than  one  event.  If  you 
require  it  in  several  successive  events,  the  observer  can  watch  for  consistency  in  performance, 
detect  trends,  allow  the  unit  to  learn  from  experience,  or  directly  coach  the  unit  on  how  to 
improve.  Avoid  the  use  of  "global  tasks"  (i.e.,  tasks  that  are  performed  and  that  the  observer  is  to 
watch  for  at  all  times  rather  than  at  specific  times).  The  most  likely  result  is  that  they  will  not  be 
observed  at  all.  Instead,  focus  on  a  few  places  where  the  tasks  can  best  be  observed. 

On  completion  of  this  activity,  you  will  have  a  draft  of  all  of  the  design  documents  for  the 
exercise  and  segments.  This  is  the  information  that  forms  the  basis  for  the  development  of  the 
exercise  package  components  in  Phase  4.  If  you  discover,  as  you  develop  the  exercise 
components,  that  need  to  change  or  rearrange  the  events,  you  should  revise  your  outline 
accordingly.  It  is  a  document  for  the  record,  and  keeping  everything  current  and  in  agreement 
will  help  you  in  making  all  of  the  products  in  Phase  4  agree. 

If  you  ever  have  to  revise  these  events,  make  sure  that  you  go  back  and  check  the  other 
elements  to  verify  the  correctness  of  all  of  the  pieces.  Everything  is  interrelated  now,  and  a 
change  at  any  point  can  cause  other  changes  to  cascade  through  the  entire  design. 

Formative  Evaluation 

The  formative  evaluation  for  this  activity  is  two-fold,  comprising  internal  reviews  and 
pilot  tests.  In  the  internal  review,  you  should  make  sure  that  all  of  the  pieces — mission 
narratives,  overlays,  outline  with  context  and  specifications,  events,  and  so  on — are  in  full 
agreement.  One  way  to  do  this  is  by  involving  several  interested  colleagues  who  are 
knowledgeable  about  the  unit  and  mission,  and  have  them  walk  through  the  entire  operation 
while  you  observe  and  take  notes. 

The  second  formative  evaluation  activity  is  to  conduct  another  series  of  tryouts  of  the 
exercise,  this  time  with  more  players.  These  are  usually  referred  to  as  pilots.  This  is  probably 
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the  first  time  that  you  will  bring  together  the  scenario  storyline,  the  simulation  components,  and 
live  participants.  Your  players  should  be  individuals  who  are  knowledgeable  about  the  subject 
matter  (the  mission,  unit  type,  operations,  etc.);  they  may  be  representative  of  the  target 
audience,  as  individuals  or  as  a  group  (i.e.,  each  one  has  battalion  staff  experience  or  they  are  an 
intact  battalion  staff).  If  possible,  try  to  involve  people  who  are  attuned  to  instructional  design 
considerations. 

If  you  have  been  using  the  intended  observers  as  subject  matter  experts  already,  you  can 
ask  them  to  participate  in  the  pilot  as  observers.  But  if  they  haven't  been  involved  yet,  don't 
bring  them  in  for  their  first  time  right  now  to  act  as  observers.  During  pilots,  you  want  to 
standardize  everything  except  the  live  participants  and  their  execution  attempts.  You  don’t  really 
need  “observers”  to  act  as  performance  evaluators;  you  need  formative  evaluators  who  can  help 
to  discover  whether  or  not  the  scenario  is  constructed  properly. 

Your  main  objectives  in  the  observations  in  these  pilots  are  to  determine  whether: 

•  the  scenario  and  order  are  tactically  appropriate  for  the  terrain  and  events, 

•  the  exercise  specifications  for  other  units,  friendly  and  enemy,  are  correct, 

•  the  terrain  locations  permit  the  exercise  to  unfold  as  you  intended, 

•  the  event  cues  cause  the  right  (i.e.,  intended)  things  to  happen, 

•  the  critical  performances  are  observable  by  prequalified  observers  (i.e.,  the  developers), 
and 

•  the  performance  standards  are  clearly  stated  and  achievable. 

Although  you  should  be  able  to  test  much  of  this  before  the  pilots,  you  are  now  doing  a 
more  complete  run  to  check  on  how  the  pieces  fit.  You  should  be  observing,  taking  notes,  and 
debriefing  all  of  the  participants  (players  and  controllers),  rather  than  participating.  If 
considerable  changes  are  made  as  a  result  of  a  pilot,  you  should  repeat  the  pilot  using  the  revised 
exercise. 

As  you  plan  for  and  conduct  the  pilots,  you  should  have  a  detailed  plan  for  how  they  will 
be  conducted,  who  will  be  involved,  how  you  will  obtain  information  (e.g.,  questionnaires, 
interviews),  and  how  you  plan  to  use  the  information.  There  are  numerous  publications  on 
conducting  program  evaluations.  Several  of  the  ARI-sponsored  projects  have  used  the 
Evaluator ’s  Handbook  (Herman,  Morris,  &  Fitz-Gibbon,  1987)  as  the  starting  point  in  designing 
pilot  test  plans. 
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PHASE  4.  DEVELOP  TRAINING  SUPPORT  PACKAGE 

Activity  4.1  Design  training  support  package  structure. 

Activity  4.2  Prepare  training  support  package  materials. 


This  is  the  final  phase  in  the  development  methodology,  wherein  you  will  actually 
develop  all  of  the  components  associated  with  the  structured  training  program.  To  a  great  extent, 
this  process  began  when  you  first  began  to  list  the  tasks  (Phase  2)  and  prepare  the  scenario 
materials  (Phase  3).  This  section  describes  how  to  translate  that  earlier  documentation  into  the 
final  products. 

In  order  to  do  this,  you  need  to  know  what  the  implementation  situation(s)  will  be  and 
what  the  materials  need  to  help  the  implementers  to  do.  By  “situation,”  we  mean  the  constraints 
and  resources  available  for  implementation  at  the  site  or  sites  where  the  program  will  be  used. 


i  EXAMPLE:  Defining  the  implementation  situation 

I  GIVEN:  Platoon-  and  company  level  SIMNIET-based  training;  brigade  staff  BBS-based  training 

I  The  platoon-  and  company-level  training  was  designed  for  implementation  at  one  fixed 
j  site  and  one  mobile  site  configuration,  and  users  were  to  be  Army  National  Guard  (ARNG) 

I  units.  The  brigade  staff  training,  on  the  other  hand,  was  designed  for  implementation  at 
I  the  user  units’  locations,  and  each  implementation  would  vary  in  terms  of  observer  teams, 
i  simulation  support  and  configuration,  time  available,  and  the  particular  training  needs  and 
i  desires  of  the  unit  in  training. 


In  Activity  4.1,  the  task  is  to  design  a  TSP  structure  that  addresses  all  of  the  known 
situations  for  implementation,  so  that  the  materials  developed  in  Activity  4.2  can  be  used  in  as 
many  of  those  situations  as  possible.  There  is  no  reason  to  wait  until  the  completion  of  Phase  3 
to  collect  this  information;  you  have  probably  been  accumulating  it  all  along. 
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Activity  4.1  Design  training  support  package  structure 

Product/Outcome:  Design  outline  for  TSP  components  that  considers  (at  a  minimum): 

✓  use  of  tactical  and  other  scenario  materiais, 

✓  unit  preparation  (training  audience), 

y  materials  to  aid  or  train  other  participants, 
y  simuiation  needs,  and 

✓  exercise  management. 

Formative  Evaluation;  Proponent  review;  review  by  representative  users. 


The  design  outline  for  the  TSP  is  not  a  final  product.  The  activity  is  included  in  this 
phase  in  order  to  focus  your  attention  on  the  need  for  some  careful  analysis  of  user  needs  and 
situation  variables  that  have  an  impact  on  the  utility  of  the  TSP. 

In  order  to  perform  this  activity,  you  will  need  to  know  (or  find  out  about)  the  intended 
implementation  situations.  Some  of  the  things  to  consider  and  discover  are: 

•  Tactical  and  other  Scenario  Materials: 

•  What  situational  (tactical)  materials  are  needed? 

•  When  will  the  situational  materials  be  handed  out  or  released  or  distributed? 

•  Who  will  do  distribution,  and  how  will  they  know  when  to  do  it? 

•  Who  gets  copies,  and  of  which  parts? 

•  Training  Audience: 

•  How  will  the  training  audience  be  prepared  for  the  program? 

•  Will  they  have  done  this  kind  of  program  before? 

•  How  much  time  will  they  have  for  preparation? 

•  What  will  they  already  know? 

•  What  misconceptions  are  possible  and  how  can  they  be  reduced? 

•  Do  they  have  computer  support  in  case  the  TSP  could  go  on  CD-ROM  or  in 
case  some  of  the  preparation  could  be  PC-based  multimedia? 
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•  Observers: 

•  Who  will  the  observers  be? 

•  Are  they  trained  and  experienced  as  observers,  or  does  their  experience  no 
more  than  mirror  that  of  the  training  audience? 

•  Are  they  of  at  least  peer  rank  with  the  training  audience? 

•  What  should  their  role  be  (e.g.,  coach,  mentor,  evaluator)? 

•  How  much  do  they  need  in  the  way  of  correct  performance  guides? 

•  Will  they  know  how  to  conduct  AARs? 

•  Do  they  need  AAR  job  aids? 

•  How  should  they  prepare  for  AARs? 

•  Will  they  have  time  for  extensive  AAR  preparation? 

•  Additional  Personnel: 

•  What  other  personnel  are  needed  (e.g.,  roleplayers,  interactors)? 

•  How  will  they  prepare  or  be  trained? 

•  Who  will  train  them? 

•  What  job  aids  or  guides  will  they  need? 

•  Simulation  Materials: 

•  How  will  the  simulation  be  loaded  with  the  training  scenario  tapes  or  files? 

•  Who  will  load  the  simulation  files  and  check  them? 

•  How  will  they  know  what  to  do?  What  skills  or  knowledge  will  they  already 
have,  and  what  do  you  need  to  tell  them  in  addition? 

•  Program  management/administration: 

•  Who  will  administer  or  manage  the  training? 

•  Will  the  manager(s) 

•  Have  experience  with  structured  simulation-based  programs? 

•  Be  available  for  preparation  and  execution  of  the  training? 

•  Have  assistants? 

•  Be  at  least  a  peer  (in  rank)  of  the  senior  person  in  the  training 
audience? 

•  Have  print-plant  support  (for  paper-based  TSP  materials)  and/or 
computer  support  (for  PC-based  multimedia  TSP  materials)? 

•  TSP  Packaging: 

•  How  will  the  TSP  be  packaged  and  distributed? 

These  are  just  some  of  the  considerations  that  you  will  need  to  address. 
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The  final  question  in  the  long  list  above  deserves  some  special  discussion:  How  will  the 
TSP  be  packaged  and  distributed?  As  the  developer,  you  may  actually  be  considering  two  levels 
of  packaging: 

•  On  one  level,  you  are  deciding  how  to  create  a  “shelf’  version  of  the  materials, 
sometimes  referred  to  as  “The  Box.”  This  is  the  master  set  that  serves  as  a  copy-ready 
version;  it  may  get  sent  to  an  implementation  site,  but  generally  is  not  the  set  that  gets 
distributed  to  lots  of  users. 

•  The  second  version  is  “The  Distribution  Set.”  It  consists  of  copies  of  materials  in  The 
Box  version  that  are  assembled  and  distributed  as  appropriate  to  the  exercise 
participants  and  administrators. 

As  an  organizing  structure  for  thinking  about  TSPs,  we  have  found  the  five-part  structure 
presented  in  TRADOC  Regulation  350-70  (Department  of  the  Army,  1995)  to  be  useful.  It  is 
diagrammed  in  Figure  6.  The  five  parts  are: 

•  Tactical  materials. 

•  Unit  materials. 

•  Train-the-trainer  materials. 

•  Simulation  materials. 

•  Administrative  materials. 

This  structure  is  more  akin  to  the  description  of  The  Box  than  to  The  Distribution  Set  of 
the  TSP.  In  most  situations.  The  Box  version  is  smaller  than  The  Distribution  Set.  This  is 
because  there  will  usually  need  to  be  multiple  copies  of  some  TSP  items  for  multiple  users. 

While  The  Box  might  contain  only  one  copy  of  the  item.  The  Distribution  Set  would  include 
multiple  copies. 

What  you  prepare  will  likely  be  The  Box  (the  shelf  version).  Certainly,  The  Box  could  be 
an  exact  model  of  The  Distribution  Set.  For  example,  if  three  copies  of  an  OPORD  are  needed  in 
implementation  for  three  members  of  the  training  audience,  then  The  Box  might  contain  three 
identical  copies  of  the  OPORD.  It  is  more  likely  that  it  will  contain  only  one. 

Other  items,  though,  will  almost  certainly  occur  in  multiple  copies  in  The  Box.  For 
example,  all  members  of  the  training  audience  may  get  individual  guides,  in  which  the  first 
section  is  always  the  same.  Rather  than  providing  just  one  copy  of  that  section  and  requiring  the 
administration  team  to  run  copies  and  insert  it  into  multiple  guides,  it  would  likely  be  included  as 
a  part  of  each  copy-ready  guide. 
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THE  TRAINING  SUPPORT  PACKAGE 


Tactical  materials:  * — 

•  OPORDs  and  other  mission-specific  materials 

•  Prepared  messages  and  scripted  materials 

•  Descriptions  of  personnel  and  equipment  status 


Unit  preparation  materials:  [ _ 

•  Guidance  on  exercise  selection 

•  Guidance  on  unit  preparation 

•  Description  of  time,  personnel,  and  facility  requirements 


Annex  C,  OPORD  95-53 
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Guidance  for  other  participants:  ‘ ^ 

•  Observer  materials  on  task  performance  and  feedback  sessions 

•  Instructions  for  operation  of  the  simulation  components 

•  Guidance  for  roleplayers 


Administrative  guidance  for  managers:  ti: 

•  Instructions  for  setting  up  the  training 

•  Instructions  for  controlling  scenario  events 


Simulation  tapes  and  documentation: 

•  Initialize  the  system 

•  Provide  starting  conditions 

•  Show  graphic  control  measures 
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Figure  4.  Five  components  of  the  TSP. 

Thus  the  decision  must  be  made  whether  (a)  to  provide  one  copy  of  an  item  in  The  Box 
and  require  that  various  numbers  of  copies  be  made,  or  (b)  to  provide  multiples  of  each  item  and 
require  that  one  copy  be  made  of  each  item  in  The  Box.  The  decision  will  usually  depend  on  the 
size  of  the  item  (big  items  being  provided  only  once  in  The  Box)  and  the  location  of  the  item  (if 
stand-alone,  then  provided  only  once  in  The  Box;  if  part  of  another  item,  then  provided  with  each 
of  the  other  items). 
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EXAMPLE:  Implementation  situation,  TSP  configuration,  and  comparison  between  The 
Box  and  The  Distribution  Set 

NOTE;  For  all  three  of  the  programs  described  below,  The  Box  was  provided  to  the  exercise 
implementers,  who  in  turn  prepared  The  Distribution  Set,  following  instructions  in  The  Box. 

(1)  The  Virtual  Training  Program:  Platoori-,  company-  ,  battalion-,  and  brigade-level  training, 
for  maneuver  and  fire  support  elements,  execution  only.  To  be  implemented  at  one  fixed 
site  and  2  mobile  sites,  using  a  dedicated  0/p  team.  Uses  SIMNET  or  Janus. 
Approximately  112  independent  exercises  provided.  ,  : 

Because  the  exercises  were  to  be  used  in  very  limited  situations  (essentially  one  site,  with 
a  dedicated  0/C  team  to  guide  the  user  units  during  all  planning,  preparation,  and  conduct 
of  the  exercise),  the  platoon-  and  company-level  TSP  was  easily  conceptualized  as  having 
three  parts  (i.e.,  trainer,  trainee,  and  tactical  materials) simple  organization 
and  content  are  shown  in  Figure  5. 

(2)  The  COBRAS  Brigade  Staff  Exercise:  Brigade  staff  training  for  combat,  combat  support, 
and  combat  service  support,  on  planning,  preparation,  and  execution  (including 
consolidation  and  reorganization).  For  export  to  brigade  training  sites  and  implemented 
within  brigade  or  division  resources. 

For  this  program,  there  were  many  more  likely  variations  in  impiementation,  as  well  as 
many  more  participants,  and  the  list  of  considerations  grew  accordingly.  The  TSP 
addresses  many  more  facets  of  implementation  than  did  the  platoon-  and  company-level 
product.  Figure  6  shows  the  TSP  components  for  the  brigade-level  exercise. 

(3)  The  COBRAS  Brigade  Staff  Vignettes:  Brigade  staff  small  group  training  vignettes, 
for  2-14  participants  each.  25  independent  event-focused  exercises,  requiring  mostly 
only  live  simulation.  Conduct  requires  1/2  -  1  day  each,  and  all  preparation,  control, 
and  feedback  is  coordinated  by  the  brigade  executive  officer  (XO). 

The  vignettes  were  envisioned  as  being  used  in  the  brigade’s  “Thursday  morning  staff 
development”  situations.  Thus  the  amount  of  TSP  materials  and  preparation 
requirements  was  severely  restricted,  the  vignette  TSPs  have  the  simplest  structure  of 
these  related  programs;  it  is  shown  in  Figure  7. 

But  there  is  another  alternative  that  may  require  consideration:  As  computers  and 
databases  become  more  accessible  to  users,  it  is  likely  that  The  Box  will  be  replaced  by  The  CD- 
ROM.  A  database  management  system  will  enable  the  training  manager  to  indicate 
implementation  options,  and  will  construct  the  appropriate  TSP  for  that  implementation.  The 
constructed  TSP  will  then  be  printed  or  otherwise  output  for  the  users,  as  The  Distribution  Set. 

In  this  case,  the  shelf  version  (The  CD-ROM)  should  not  contain  any  item  more  than  once. 
Rather,  the  database  links  should  be  constructed  in  such  a  way  that  the  right  items  get  added  to 
the  appropriate  sets  of  distribution  materials.  In  this  way,  corrections  or  changes  to  individual 
pieces  can  be  made  just  one  time  and  populated  throughout  the  TSP. 


32 


TSP  Components  and  Structure  for  the  Virtual  Training  Program  (VTP) 


TSP  Component 

Copies  in 
The  Box/ 
Distribution 
Set 

Distribution 

Orientation  Guide;  Overview  of  the  exercises;  description 
of  each  exercise  (of  1 1 2) 

1/1 

1  per  unit,  to  assist  in 
exercise  selection  and 
planning 

Preparation  Materials:  Overview  of  the  exercise;  general 
information  on  how  to  get  ready,  participants,  and  what 
to  expect.  Preview  and  task  demonstration  videotapes. 

1/1 

1  per  unit,  to  assist  them  in 
preparation 

Observer/Controller  Materials:  Instructions  on  how  to 
prepare  the  unit;  what  to  hand  out  and  when;  how  to 
control  the  simulation  events;  how  to  conduct 
orientation,  tactical  events,  and  AARs. 

1 /variable 

1  per  O/C  (non¬ 
expendable) 

Tactical  Materials  (one  volume  per  mission);  OPORDs  for 
the  user-level  unit  and  the  higher  level  unit;  overlays; 
exercise  intent  narratives;  exercise  event  guides 
showing  O/C  actions,  simulation  activity,  unit  responses 
expected,  training  objectives;  exercise-specific  AAR 
worksheets,  job  aids,  and  take  home  package 
worksheets. 

1 /variable 

1  set  of  orders  and 
overlays,  specific  to  the 
selected  exercises,  for  the 
user  unit,  to  be  used  in 
preparation. 

1  set  of  all  contents, 
specific  to  the  selected 
exercises,  for  the  O/Cs 
assigned  to  that  execution. 
Most  parts  expended 
during  the  exercise. 

Number  of  0/Cs  required 
for  different  exercises 
varies  (between  1  and  10). 

Simulation  Materials:  Simulation  documentation  and 
tapes. 

1/1 

For  use  in  initializing  the 
appropriate  system  for 
each  exercise. 

Figure  5.  TSP  structure  and  distribution  plan  for  the  Virtual  Training  Program  (VTP). 
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1  TSP  Components  and  Structure  for  the  COBRAS  Brigade  Staff  Exercise  (page  1  of  2) 

TSP  Component 

Copies  in 
The  Box/ 
Distribution 
Set 

Distribution 

Orientation  Guide;  Overview  of  the  exercises; 
description  of  each  exercise,  how  to  initiate 
preparation. 

1/1 

1  per  unit,  to  assist  in  exercise 
selection  and  planning. 

Training  Audience  Preparation  Materials; 

Overview  of  the  exercise;  general 
information  on  how  to  get  ready, 
participants,  and  what  to  expect;  list  of 
training  objectives  for  each  primary  training 
audience  member. 

1/16 

1  per  primary  training  audience 
member,  with  the  specific  task  list  for 
that  individual  and  the  selected 
mission. 

Exercise  Director,  COBRAS  Coordinator,  and 
BLUFOR  Controller  Materials;  Instructions 
on  how  to  plan  for  the  exercise;  working  with 
the  brigade  to  select  implementation  options; 
working  with  the  simulation  site  to  initiate 
preparation  and  implement  the  exercise; 
required  resources;  training  participants 
(roleplayers  and  interactors);  what  to  copy 
and  distribute,  and  when;  how  to  control  the 
simulation  and  other  events  for  each 
mission. 

1/3 

1  per  person,  to  guide  them  in 
preparation  and  during  the  exercise. 

Roleplayer  Materials;  Specifics  of  the  unit  they 
represent;  tactical  and  training  intent  for  the 
exercise  and  each  mission;  rules  of 
engagement;  using  the  simulation.  Division 
response  cell  guide  contains  scripted  (radio) 
and  Tactical  Fax  (TACFAX)  messages. 

1  each/ 

1  each 

1  per  roleplayer  cell,  to  be  used 
during  preparation,  exercise 
rehearsal,  and  exercise  conduct.  12 
role-specific  guides,  for  division 
response  cell,  subordinate  units, 
supporting  CS  and  CSS  units,  and 
OPFOR. 

Observer  Materials;  Overview  of  the  exercise; 
general  information  on  how  to  get  ready, 
how  to  coach,  provide  feedback,  and 
conduct  the  AAR  (Senior  Observer  only);  list 
of  training  objectives  for  the  primary  training 
audience  member  each  will  observe,  for  the 
selected  mission. 

1 /variable 

1  per  observer,  with  the  specific  task 
list(s)  for  that  individual  and  the 
selected  mission.  Between  1 1  and 

16  observers  required,  depending  on 
availability  of  experts  in  the  different 
functions  to  be  tapped  as  observers. 

AAR  Materials;  Mission-specific  guidance  and 
slides  to  be  used  in  AARs 

1/1 

For  use  by  the  Senior  Observer  in 
conducting  the  brigade  AARs.  Time 
for  AAR  preparation  is  constrained  to 
about  one  hour,  up  to  8  AARs  per 
mission  may  be  conducted. 

Figure  6.  TSP  structure  and  distribution  plan  for  the  COBRAS  Brigade  Staff  Exercise. 


Continued  on  next  page 
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1  TSP  Components  and  Structure  for  the  COBRAS  Brigade  Staff  Exercise  (page  2  of  2) 

TSP  Component 

Copies  in 
The  Box/ 
Distribution 
Set 

Distribution 

Sample  Brigade  Products;  For  use  by 
observers  in  illustrating  general  form  and 
content  of  brigade  planning  products. 

1/1 

Available  for  use  by  observers. 

Tactical  Materials 

OPORDs  for  the  corps  and  division; 
overlays;  intelligence  products. 

1/5 

Specific  to  the  selected  mission. 

2  sets  of  orders  and  overlays  for  the 
brigade,  handed  out  as  first  activity  of 
the  exercise 

1  set  of  orders  each  for  observer 
team,  division  response  cell,  and 
Exercise  Director,  to  be  used  in 
preparation 

1  set  of  overlays  for  division 
response  cell 

1  set  of  intelligence  products  for 
division  response  cell,  for 
dissemination  to  the  brigade 
according  to  the  tactical  situation. 

Current  situation  descriptions  and  unit 
readiness  levels;  mission  specific. 

1 /variable 

Mission-  and  unit-specific 
descriptions  for  each  roleplayer  of 
subordinate  and  supporting  units,  for 
each  training  audience  member  and 
observer. 

Simulation  Materials;  Basic  TO&E  and 
initialization  tapes  for  Brigade/Battalion 

Battle  Simulation  (BBS),  complete 
documentation  of  tape  data. 

1/1 

Simulation  center;  appropriate 
mission-specific  tapes  loaded  to 
initialize  exercise. 

Documentation  available  to  permit 
modifications,  to  be  used  in  case  of 
incompatible  simulation  upgrades. 

Interactor  Materials:  How  to  use  the  simulation 
to  accomplish  required  tasks  in  controlling 
units. 

1  of  each/ 
variable 

1  of  the  appropriate  guide  for  each 

BBS  workstation;  9  (BLUFOR),  2 
(OPFOR),  1  (EXCON  -  Division). 

Simulation  Center  Materials:  Instructions  and 
guidance  for  simulation  center  personnel  on 
initialization  and  control  of  the  exercise, 
other  participants,  training  interactors. 

1/1 

Simulations  Center  manager,  for  use 
in  planning,  preparation,  and  conduct 
of  exercises. 

Figure  6.  TSP  structure  and  distribution  plan  for  the  COBRAS  Brigade  Staff  Exercise 


(continued). 


1  TSP  Components  and  Structure  for  the  COBRAS  Brigade  Staff  Vignettes 

TSP  Component 

Copies  in 
The  Box/ 
Distribution 
Set 

Distribution 

Guide  to  Use  and  Implementation  of 

COBRAS  Vignettes;  Overview  of  the 
vignette  intents;  how  to  select,  prepare,  and 
conduct. 

1/1 

1  per  unit,  to  assist  in  vignette 
selection  and  planning. 

Training  Coordinator  Materials:  Overview  of 
the  vignette  scope,  participants,  and  tasks; 
information  on  how  to  get  ready;  list  of 
training  objectives;  how  to  initiate  and 
control  the  vignette;  AAR  questions. 

1  each/ 1 
each 

1  per  vignette  (of  25). 

Training  Participant  Materials:  Overview  of 
the  vignette  scope  and  tasks;  information  on 
how  to  get  ready;  list  of  training  objectives 
and  references. 

1 /variable 

1  per  participant  (vignette-specific); 
participants  range  from  2  to  14. 

Preparation  Materials:  Selected  tactical 

materials  to  provide  the  setting  and  situation 
for  the  vignette  problem. 

1 /variable 

According  to  vignette-specific 
instructions;  used  by  participants  to 
prepare  for  the  vignette  conduct. 

Execution  Materials;  Selected  tactical  materials 
to  cue  and  shape  the  vignette  problem. 

1 /variable 

According  to  vignette-specific 
instructions;  used  by  participants 
during  vignette  conduct. 

Job  Aid  Materials;  Provided  for  selected 
vignettes  to  help  participants  perform  the 
tasks. 

1 /variable 

According  to  vignette-specific 
instructions;  used  by  participants 
during  vignette  conduct. 

Sample  Products:  For  use  to  illustrating 
general  form  and  content  of  brigade  staff 
products. 

1/1 

Vignette-specific;  available  for  use  by 
Training  Coordinator. 

Support  Coordinator  Materials:  For  use  in 

simulation-supported  vignettes;  guidance  for 
roleplayers  and  interactors;  simulation  tapes 
and  documentation. 

1/1 

Guide  and  instructions  to  Support 
Coordinator  for  distribution  to 
roleplayers  and  interactors. 

Tapes  and  documentation  to 

Simulation  Center  Manager. 

Figure  7.  TSP  structure  and  distribution  plan  for  the  COBRAS  Brigade  Staff  Vignettes. 
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Formative  Evaluation 


As  with  the  interim  product  in  Phase  1,  the  facts  and  assumptions  that  you  document 
about  implementation  should  be  verified  by  your  proponent.  Additionally,  an  early  check  on 
your  proposed  design  by  some  potential  users  will  give  you  valuable  information.  The  most 
useful  feedback  will  likely  come  from  leaders  and  training  officers  of  units  who  would  use  the 
materials  and  from  simulation  site  managers. 


Activity  4.2  Prepare  training  support  package  (TSP)  materiais 
Product/Outcome:  The  TSP,  as  outlined  in  Activity  4.1 
Formative  Evaluation;  Expert  reviews,  trials  with  representative  personnel 


This  activity  makes  use  of  all  of  the  information  and  development  work  that  has  gone  on 
to  date.  This  is  the  culmination  of  the  development,  leading  up  to  the  implementation. 

If,  however,  you  discover  any  discrepancies  between  what  should  go  into  the  TSP  and 
what  you  have  available,  then  you  may  need  to  revisit  one  or  more  of  the  earlier  activities. 
Sometimes  you'll  just  engineer  the  missing  piece  of  material;  other  times  you  may  need  to  make 
some  more  sweeping  changes  in  order  to  maintain  the  correspondence  among  all  of  the  package 
components. 

The  most  important  considerations  that  affect  your  efforts  for  this  activity  are  the  typical 
instructional  design  considerations:  to  make  the  materials  not  only  technically  correct  and 
complete,  but  also  situation-  and  user-friendly.  The  outline  for  TSP  that  you  designed  in  Activity 
4. 1  serves  as  your  blueprint  here. 

Broadly  defined,  the  TSP  is  everything  that  is  needed  to  support  every  aspect  of 
implementation  of  the  program.  Every  TSP  will  be  different,  because  every  TSP  is  specific  to 
the  particular  training  program  it  supports.  The  remainder  of  this  section  contains  some 
suggestions  drawn  from  our  experience  about  how  to  prepare  a  TSP.  But  you  should  be  ready  to 
improvise  as  necessary  so  that  the  TSP  you  develop  is  appropriate  for  the  training  you  develop. 

If  you  are  an  instructional  developer,  then  you  understand  the  principles  of  materials 
development  that  can  make  the  program  accessible;  if  not,  the  points  below  will  highlight  some 
important  things  that  you  can  do  to  make  sure  that  the  users  can,  in  fact,  use  the  materials. 
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•  The  organization  of  the  materials  has  to  be  precise  and  clear.  The  user  must  be  able 
to  find  the  critical  information  quickly.  While  it  may  also  be  important  to  include 
background,  explanation,  justification,  and  so  on,  the  critical  how-to  information  must 
be  clearly  visible.  Think  about  the  difference  between  “just  in  case”  presentation  that 
includes  all  of  the  whys  and  wherefores,  and  “just  in  time”  presentation  that  is 
instructional  and  directive,  with  a  separate  section  of  reasons  why. 

•  Don’t  be  bound  by  too  many  traditional  ideas  of  how  training  materials  should  look. 
You’re  writing  for  users,  and  the  desktop  publishing  people  have  done  a  lot  to  help 
specify  what  it  takes  to  make  materials  useful.  You  should  be  aiming  for  materials 
that  are  easy  to  read  and  understand,  using  graphics  where  they  aid  in  understanding. 
You  are  not  writing  a  textbook  (probably).  But,  of  course,  be  aware  of  any  layout  and 
format  restrictions  that  are  already  in  place. 

•  Some  of  the  user  guides  put  out  by  computer  manufacturers  and  software  writers 
serve  as  useful  models.  Spend  some  time  looking  at  them  for  ideas,  or  look  at  the 
VTP  or  COBRAS  TSP  materials.  You’ll  probably  notice  lots  of  white  space,  use  of 
graphics  and  look-up  tables,  icons  and  symbols  that  catch  the  user’s  attention,  lines  or 
borders  that  help  to  group  relevant  material,  headers  and  footers  that  help  the  user 
know  where  he/she  is,  and  so  on. 

•  Use  job  aids.  These  will  help  the  user  know  what  he  has  to  learn  beforehand,  and 
what  he  can  look  up  when  necessary. 


EXAMPLE:  Usingjobaids 

Some  Job  aids  that  have  been  useful  in  currently  used  programs  include; 

•  A  planning  and  preparation  timeline  of  events  leading  up  to  the  exercise 

•  A  list  of  the  personnel  to  be  tasked 

•  A  chart  showing  the  materials  in  the  fSP  and  who  gets  what  , 

•  A  diagram  of  a  likely  simulation  site  layout 

•  An  event  list  of  the  scenario  storyline  for  the  exercise,  showing  both  higher  echelon 
and  OPFOR  event  that  cue  th’e  unit  tasks 

•  A  chart  of  scripted  messages  for  use  by  the  higher  echalon  response  cell 


•  As  you  package  the  materials  into  The  Box  or  The  CD-ROM,  consider  preparing  a 
“Read  Me  First”  document  that  will  point  to  the  rest  of  the  TSP,  so  that  the  training 
manager  knows  where  to  start. 
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Formative  Evaluation 


The  final  formative  evaluation  is  more  formal  than  the  earlier  reviews.  In  order  to  insure 
that  all  of  the  materials  are  both  (a)  correct  and  (b)  user-friendly,  two  formative  evaluation  steps 
should  be  initiated: 

•  Expert  Review.  Expert  reviews  of  the  simulation-connected  parts,  the  training  content, 
and  the  instructional  design  will  insure  face  validity  as  well  as  helping  you  to  refine  all 
of  the  components  of  the  training  program  and  package.  There  are  several  levels  of 
expert  review  recommended: 

•  Individuals  who  are  experts  on  the  simulation  and  its  components  should  make 
sure  that  your  exercise  materials  are  appropriate  for  the  technology;  they  should 
pay  particular  attention  to  the  materials  used  to  program  or  configure  the 
simulation  for  the  scenario. 

•  Individuals  who  are  experts  on  the  doctrinal  issues  for  the  mission  should 
review  the  materials  to  be  used  during  orientation,  execution,  and  the  AAR  for 
doctrinal  accuracy. 

•  Someone  who  is  familiar  with  the  needs  and  resource  constraints  of  the  target 
units  should  check  the  advance  materials  and  program  management  materials. 

•  A  review  by  instructional  design  experts  can  help  you  with  the  presentation  of 
all  of  the  information. 

•  An  internal  roleplay  of  the  unpacking,  reproducing,  and  distributing  processes, 
using  the  instructions  in  the  TSP,  will  help  to  catch  disconnects  in  the 
instructions. 

•  Trial(s).  Trials  that  involve  representative  personnel  acting  as  participants  will  provide 
evidence  of  the  usability  of  the  materials.  Data  collection  will  be  standardized  and 
intensive,  involving  individual  and  group  interviews  and  detailed  observations  of  a 
wide  variety  of  indicators.  As  with  the  pilots,  you  should  have  a  formal  plan  for  the 
trials.  Some  of  the  important  considerations  include: 

•  Involve  actual  units  and  other  target  personnel  in  as  many  of  the  roles  as 
possible  (in  contrast  to  the  pilots  in  Activity  3.3,  where  you  may  have  used 
surrogate  participants). 

•  Use  TSP  materials  that  are  as  close  to  “perfect”  as  you  can  get;  don’t  plan  on 
using  available  developers  to  interpret  the  materials  to  the  users.  You  want  the 
users  to  diligently  attempt  to  use  the  materials  as  they  are  intended  to  be  used. 

•  However,  you  may  also  need  to  interrupt  the  exercise  at  times  to  determine  what 
went  wrong  or  to  work  around  a  fatal  error. 
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•  During  exercise  preparation  and  the  conduct  of  the  exercise  itself,  collect  data: 
observations,  opinions  and  suggestions,  discussions  of  ways  of  fixing  things  that 
aren't  yet  right.  After  the  training,  you  will  interview  the  participants  to  get 
their  reactions  to  the  training  content  and  materials. 

•  Use  a  variety  of  methods  to  collects  the  information:  observations,  group 
discussions,  structured  interviews,  questionnaires  and  surveys,  video-  and 
audio-tapes. 

•  The  aspects  of  the  program  that  are  best  examined  during  a  trial  include 
verification  of  assumptions  and  expectations,  assessment  or  refinement  of  the 
methods  for  conducting  the  exercise,  evaluation  of  the  clarity  and  utility  of  the 
materials,  and  judgment  of  the  completeness  of  the  materials  and  absence  of 
extraneous  material. 

•  Document  your  formative  evaluation  trial  plan:  participants,  process,  data 
collection  methods,  and  any  caveats  or  alibis  about  how  the  trial  is  conducted 
and  how  that  differs  from  the  intended  implementation. 

If  you  expect  to  have  a  dedicated  observer/controller  team  during  the  actual 
implementation,  you  should  try  to  involve  them  also  in  the  trials.  You  will  need  to  train  them  in 
the  roles  and  responsibilities  specific  to  these  exercises.  Section  4  of  the  original  methodology 
guide  (Campbell,  Campbell,  Sanders,  Flynn,  &  Myers,  1995)  discusses  0/C  training  at  length. 

These  two  kinds  of  formative  evaluation  processes  (expert  reviews  and  trials)  should  be 
done  for  every  component  of  the  TSP.  All  of  the  information  is  collected  for  the  purpose  of 
revising  the  materials.  Ideally,  once  materials  are  revised,  another  trial  should  be  conducted, 
until  all  of  the  flaws  are  worked  out  or  until  your  resources  and  time  are  expended. 
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APPENDIX 


Guide  to  Development  of 
Structured  Simulation-Based  Training 

OUTLINE  OF  THE  METHODOLOGY 


Phase  1.  Initial  Decisions 


Activity  1 . 


Product  or 
outcome 


Formative 

evaluation 


Decision  areas 
(pages  7-10 
of  the  Guide) 


Document  initial  Decisions 


Documentation  of  decisions 


Proponent  review 


1 .  Target  training  audience 

2.  Training  context 

•  Mission  type 

•  Enemy  type 

•  Terrain/locale 

•  Unit  type 

3.  Simulation  technology 

4.  Other,  e.g.: 

•  Exercise  time 

•  Number  of  entry  points 

•  Nature  of  entry  points 

•  Linkages  to  other  programs 

•  Guidance  for  training  priorities 

•  Trainer  and  other  resources 
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Phase  2.  Designate  Training  Objectives 


NOTE 

Either  Activity  2.2  or  Activity  2.3  can  be  done  first.  Do  whichever  you 
think  will  reduce  the  task  list  most. 

NOTE 

Phase  2  and  Phase  3  (designing  the  scenario  and  exercise  structure)  will 

probably  be  iterative. 

Activity  2.1  Identify  Task  Sources,  Tasks,  and  Standards 


Product  or  Tasks  and  standards;  task  sources 

outcome 


Formative  Internal  and  proponent  review 
evaluation 


Sources 
(pages  12-13 
of  the  Guide) 


•  Doctrinal  —  ARTEP-MTPs,  FMs,  proponent  agency  lists 

•  Other — job  and  task  analysis 


Task  detailing 
(page  13 
of  the  Guide) 


•  Task  statement  —  lowest  level  of  collective  behavior  that  has 
accompanying  conditions  and  standards 

•  Conditions  statement  —  description  of  situation,  environment,  and 
initiating  cues  that  should  cause  a  task  to  be  performed 

•  Standard  —  statement  of  correct,  acceptable,  ideal  accomplishment  of 
a  task 
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Activity  2.2  Refine  Task  List  for  Simulation  Support 


Product  or  Task  list  annotated  to  show  tasks  that  can  be  fully  or  partially  performed  and 
outcome  observed  in  the  simulation 


Formative  Internal  and  proponent  review 
evaluation 


Judgments  of 
simulation 
support 
(pages  14-15 
of  the  Guide) 


•  Use  a  rule-based  system 

•  Use  more  than  one  judge;  reconcile  judgments 

•  Use  task  expert  and  simulation  expert 

•  Document  partial  trainability  (which  parts) 

•  Determine  whether  the  task: 

•  Can  be  performed  in  simulation 

•  Should  be  trained  in  simulation  setting 

•  Can  be  performed  in  the  simulation  setting,  though  not 
simulation-supported 


Activity  2.3  Select  Tasks  That  Support  the  Mission 


Product  or 
outcome 


Reduced  task  list,  annotated  to  show  the  tasks  (or  parts  of  tasks)  that  will  be 
performed  and  can  be  observed  in  the  context  of  the  mission,  along  with 
appropriate  conditions  and  standards 


Formative  Proponent  review;  training  and  doctrine  agency  review 
evaluation 


Selecting  tasks 
(and  standards) 
(page  16 
of  the  Guide) 


•  Are  part  of  the  mission(s)  to  be  training  (per  Activity  1) 

•  Can  and  should  be  trained  in  the  simulation,  can  be  observed 

•  “Other”  guidance  for  training  priorities  satisfied 
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Phase  3.  Design  Scenario  and  Exercise  Structure 


I*!'*  NOTE  Phase  2  and  Phase  3  (designing  the  scenario  and  exercise  structure)  will 
probably  be  iterative. 

Phase  3  activities  are  closely  tied  and  highly  interdependent.  Compare  unit’s 
and  higher  echelon’s  missions  and  scenario  outlines  frequently  to  ensure 
tactical  correspondence. 


Task  sequencing 
(pages  17-18 
of  the  Guide) 


Common  sequencing  approaches: 

•  Crawl-walk-run 

•  Natural  order 

•  Hierarchical 

•  Easy-to-difficult 


Activity  3.1  Design  the  Scenario 


Product  or 
outcome 


Draft  of  the  "concept  of  the  operation"  with  sketch  of  graphic  overlay,  draft  of 
unit  OPORD,  or  other  method  of  presentation;  for  training  unit  and  higher 
echelons  as  necessary 


Formative  Map  exercises 

evaluation 


Continued  on  next  page 
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Activity  3.1  Design  the  Scenario,  continued 


Design 
principles 
(page  19 
of  the  Guide) 


1 .  Situation  should  be  realistic 

•  Cues  match  real-world  conditions 

•  Tactical  materials  resemble  actual  materials  (1-2  levels  up) 

•  Plausible  activities  at  higher  echelons 

2.  Enemy  should  be  realistic 

•  Organization,  equipment,  tactics,  techniques,  procedures 
consistent  with  its  doctrine 

3.  Real-time,  real-space  events 

•  Minimal  magic  moves  and  reconstitution 

•  Time  only  moves  forward 

•  Constant  terrain 

•  Partitions  at  natural  breaks 


Components  of 
mission  outline 
(page  20 
of  the  Guide) 


1 .  Initial  locations  (friendly  and  enemy) 

2.  Major  events  list 

3.  Participants  (actual  and  notional,  friendly  and  enemy) 

4.  Terrain  area  or  enviromnent 

5.  Task  by  event  crosswalk 


Battle-oriented 
exercise 
products 
(page  20 
of  the  Guide) 


•  Description  of  mission  and  commander’s  intent 

•  Enemy  plan  (locations,  objectives,  mission,  intent,  etc.) 

•  Terrain  to  support  mission,  sketch  of  graphic  control  measures 

•  Scenario  timeline  showing  friendly  and  enemy  activities 
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Activity  3.2  Prepare  Exercise  Context  and  Specifications 


Product  or 
outcome 


Formative 

evaluation 


Product 
description 
(page  22-23 
of  the  Guide) 


Context,  specifications,  and  execution  details  for  exercises 


Simulation-controlled  exercise  to  verify  the  following: 

•  Match  between  specifications  and  simulation 

•  Suitability  of  terrain 

•  Exercise  length 

•  Correlation  of  forces 


1 .  Narrative/graphic  representation 

2.  Friendly  and  enemy  situations 

•  Equipment/persoimel  status 

•  Recent  events 

•  Starting  locations  (approximate) 

3.  Unit  specifications  at  starting  point 

•  Unit  identifiers  and  type  unit,  how  represented 

•  Initial  status  of  each  system 

4.  Exercise  ending  point — approximate  location  or  event 

5.  Scenario  description — exercise  intent,  i.e.,  overall  objective,  general 
statement  of  conditions 

6.  Tasks/training  objectives 


A-6 


Activity  3.3  Outline  Events  and  Build  Exercise 


Product  or 
outcome 


Formative 

evaluation 

(page  25-26  of 
the  Guide) 


Elements 
(page  24 
of  the  Guide) 


Master  event  list  showing  cues,  expected  performance,  and  critical  tasks  or 
subtasks;  simulation  files 


Pilot  with  knowledgeable  personnel  to  check  the  following: 

•  Scenario  and  order  are  appropriate  for  terrain  and  events 

•  Exercise  specifications  for  all  units  are  correct 

•  Locations  permit  exercise  to  proceed  as  planned 

•  Event  cues  cause  tasks  to  happen 

•  Performance  is  observable 

•  Performance  standards  are  clear  and  observable 


•  Cue  to  start  event 

•  Unit  response 

•  Task  to  be  observed 


Phase  4.  Develop  Training  Support  Package  (TSP) 


wr  note  TSPs  are  unique  to  the  training  programs  they  support. 


Necessary  inputs 


•  What  will  the  implementation  setting  be? 

•  Who  are  the  users  (implementers  and  training  unit)? 

•  How  will  they  use  the  materials? 

•  What  should  the  materials  help  them  do? 


Activity  4.1  Design  TSP  Structure 


Product  or 
outcome 


Design  outline  for  TSP  components  that  considers  (at  a  minimum): 

•  Use  of  tactical  and  other  scenario  materials 

•  Unit  preparation  (training  audience) 

•  Materials  to  aid  or  train  other  participants 

•  Simulation  needs 

•  Exercise  management 

•  TSP  packaging 


Formative  Proponent  review  and  review  by  representative  users 
evaluation 


Continued  on  next  page 
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Activity  4.1  Design  TSP  Structure,  Continued 


Considerations 
(expanded  list) 
(pages  28-29 
of  the  Guide) 


1.  Tactical  and  other  scenario  materials 

•  What  is  needed,  when  is  it  distributed,  who  gets  it,  how  do  they 
know  what  to  do  with  it? 

2.  Unit  preparation  (training  audience) 

•  Prior  experience,  current  skill  level,  time  available  for 
preparation,  what  preparation  is  needed,  what  media  for  providing 
information? 

3.  Materials  to  aid  or  train  other  participants 

•  Who  are  the  other  participants  (observers,  roleplayers, 
interactors),  prior  experience,  functions  or  roles,  what  job  aids  can 
be  used? 

4.  Simulation  needs 

•  How  will  scenario  be  transferred  to  simulation,  who  does  it,  what 
instructions  needed? 

5.  Exercise  management 

•  Who  will  manage  the  training,  prior  experience,  resources? 

6.  TSP  packaging 

•  How  will  the  TSP  be  packaged  and  distributed? 

•  Be  aware  of  differences  between  the  shelf  (master)  copy  and 
distribution  copy. 
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Activity  4.2  Prepare  TSP  Materials 


Product  or 
outcome 


Formative 

evaluation 


TSP  principles 
(page  38 
of  the  Guide) 


Expert  review 
guidelines 
(page  39  of  the 
Guide) 


The  TSP,  as  outlined  in  Activity  4.1. 


Expert  reviews,  trials  with  representative  personnel 


•  Clear  organization 

•  Easily  readable  and  understandable  appearance 

•  Timely  use  of  graphics  and  tables 

•  Extensive  use  of  appropriate  job  aids 


•  Simulation  experts  for  materials  used  to  program  or  configure 
simulation 

•  Doctrinal  experts  for  tactical  materials  and  AAR  materials 

•  Someone  familiar  with  training  needs  and  constraints  for  advance 
materials  (unit  preparation)  and  management  materials 

•  Instructional  design  experts  for  structure  and  presentation  mode 

•  Someone  for  roleplay  of  unpacking,  reproducing,  distributing 

Continued  on  next  page 
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Activity  4.2  Prepare  TSP  Materials,  Continued 


Trial  guidelines 
(page  39-40 
of  the  Guide) 


1 .  Use  actual  (representative)  participants  in  all  positions,  if  possible 

2.  Use  actual  TSP  materials 

3.  Let  participants  use  materials;  intervene  only  in  case  of  fatal  errors 

4.  Collect  information  via 

•  Observations 

•  Group  discussions  before,  during,  after  exercise 

•  Structured  interviews 

•  Questionnaires 

5.  Focus  on 

•  Assumptions  and  expectations 

•  Methods  for  conducting  exercise 

•  Clarity  and  utility  of  materials 

•  Completeness  of  materials  and  absence  of  extraneous  materials 

6.  Have  a  formal  plan 
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